<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Cerebral Mastication &#187; aws</title>
	<atom:link href="http://www.cerebralmastication.com/tag/aws/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.cerebralmastication.com</link>
	<description>Something to Chew On</description>
	<lastBuildDate>Wed, 07 Dec 2011 13:08:46 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Shell scripting EC2 for fun and profit</title>
		<link>http://www.cerebralmastication.com/2011/05/shell-scripting-ec2-for-fun-and-profit/</link>
		<comments>http://www.cerebralmastication.com/2011/05/shell-scripting-ec2-for-fun-and-profit/#comments</comments>
		<pubDate>Fri, 06 May 2011 20:57:40 +0000</pubDate>
		<dc:creator>JD Long</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[aws]]></category>
		<category><![CDATA[ec2]]></category>
		<category><![CDATA[howto]]></category>
		<category><![CDATA[R]]></category>

		<guid isPermaLink="false">http://www.cerebralmastication.com/?p=993</guid>
		<description><![CDATA[Lately I&#8217;ve been doing some work with creating ad-hoc clusters of EC2 machines. My ultimate goal is to create a simple way to spin up a cluster of EC2 machines for use with Bryan Lewis&#8217;s very cool doRedis backend for the R foreach package. But that&#8217;s a whole other post. What I was scratching my [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.thinkgeek.com/tshirts-apparel/unisex/frustrations/374d/" onclick="pageTracker._trackPageview('/outgoing/www.thinkgeek.com/tshirts-apparel/unisex/frustrations/374d/?referer=');"><img class="alignleft size-full wp-image-994" style="border: 1px solid black; margin: 2px;" title="lg-go-away-tshirt" src="http://www.cerebralmastication.com/wp-content/uploads/2011/05/lg-go-away-tshirt.jpg" alt="" width="179" height="218" /></a>Lately I&#8217;ve been doing some work with creating ad-hoc clusters of EC2 machines. My ultimate goal is to create a simple way to spin up a cluster of EC2 machines for use with Bryan Lewis&#8217;s very cool <a href="http://cran.r-project.org/web/packages/doRedis/index.html" onclick="pageTracker._trackPageview('/outgoing/cran.r-project.org/web/packages/doRedis/index.html?referer=');">doRedis backend</a> for the R <a href="http://cran.r-project.org/web/packages/foreach/index.html" onclick="pageTracker._trackPageview('/outgoing/cran.r-project.org/web/packages/foreach/index.html?referer=');">foreach package</a>. But that&#8217;s a whole other post. What I was scratching my head about today was that I&#8217;d really just like to, with a single command, spin up an EC2 instance, wait for it to come up, and then ssh into it. I do this iteration about 20 times a day when I&#8217;m testing things, so it seemed to make sense to shell script it.<br />
To do this, one needs the EC2 command line tools installed on your workstation. In Ubuntu that&#8217;s as easy as `sudo apt-get ec2-api-tools`</p>
<p>So here&#8217;s a short shell script to spin up an instance, wait 30 seconds, then connect:<br />
<script src="http://gist.github.com/959780.js"></script></p>
<p>If you&#8217;re reading this through an RSS reader, you can see the script over at <a href="https://gist.github.com/959780" onclick="pageTracker._trackPageview('/outgoing/gist.github.com/959780?referer=');">github</a>.</p>
<p>Obviously you&#8217;ll need to change the parameters at the top of the script to suit your needs. But since this was a bit of a pain in the donkey hole for me to figure out, I thought I would share.</p>
<p>If you want to help out, I&#8217;d love you to enlighten me on how to have the script figure out if an instance has finished booting so I could eliminate the sleep step.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.cerebralmastication.com/2011/05/shell-scripting-ec2-for-fun-and-profit/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
		</item>
		<item>
		<title>Controlling Amazon Web Services using rJava and the AWS Java SDK</title>
		<link>http://www.cerebralmastication.com/2010/11/controlling-amazon-web-services-using-rjava-and-the-aws-java-sdk/</link>
		<comments>http://www.cerebralmastication.com/2010/11/controlling-amazon-web-services-using-rjava-and-the-aws-java-sdk/#comments</comments>
		<pubDate>Tue, 30 Nov 2010 19:51:17 +0000</pubDate>
		<dc:creator>JD Long</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[aws]]></category>
		<category><![CDATA[R]]></category>
		<category><![CDATA[rJava]]></category>
		<category><![CDATA[S3]]></category>

		<guid isPermaLink="false">http://www.cerebralmastication.com/?p=937</guid>
		<description><![CDATA[ I&#8217;ve been messing around with using Amazon Web Services for a while. I&#8217;ve had some projects where I wanted to upload files to S3 or fire off EMR jobs. I&#8217;ve been controlling AWS services using a hodgepodge of command line tools and the R system() function to call the tools from the command line. [...]]]></description>
			<content:encoded><![CDATA[<p><img class="alignnone" style="border: 1px solid black; margin: 4px;" title="aws" src="http://awsmedia.s3.amazonaws.com/logo_aws.gif" alt="" width="164" height="60" /> I&#8217;ve been messing around with using Amazon Web Services for a while. I&#8217;ve had some projects where I wanted to upload files to S3 or fire off EMR jobs. I&#8217;ve been controlling AWS services using a hodgepodge of command line tools and the R system() function to call the tools from the command line. This has some real disadvantages, however. Using the command line tools means each tool has to be configured individually which is painful on a new machine. It&#8217;s also much harder to roll my R code up into a CRAN package because I have to check dependencies on the command line tools and ensure that the user has properly configured each tool. Clearly a pain in the ass.</p>
<p>So I was looking for more simple/elegant solutions. After thinking the <a href="http://code.google.com/p/boto/" onclick="pageTracker._trackPageview('/outgoing/code.google.com/p/boto/?referer=');">Boto</a> library for Python might be helpful, I realized that the easiest way to use that would be with <a href="http://rjython.r-forge.r-project.org/" onclick="pageTracker._trackPageview('/outgoing/rjython.r-forge.r-project.org/?referer=');">rJython</a> which meant having to interact with R, Python, AND Java. Considering I don&#8217;t program in Python or Java, that seemed like a fair bit of complexity. Then I realized that the canonical implementation of the AWS API was the AWS Java SDK. The <a href="http://www.rforge.net/rJava/" onclick="pageTracker._trackPageview('/outgoing/www.rforge.net/rJava/?referer=');">rJava</a> package makes interacting with Java from R a viable option.</p>
<p>Since I&#8217;ve never written a single line of Java code in my pathetic life, this was somewhat harder than it could have been. But with some help from <a href="http://romainfrancois.blog.free.fr/" onclick="pageTracker._trackPageview('/outgoing/romainfrancois.blog.free.fr/?referer=');">Romain Francois</a> I was able to cobble together &#8220;something that works.&#8221; The code below gives a simple example of interfacing with S3. The example will look to see if a given bucket exists on S3, if not it will create the bucket. Then it will upload a single file from your PC into that bucket. You will have to <a href="http://aws.amazon.com/sdkforjava/" onclick="pageTracker._trackPageview('/outgoing/aws.amazon.com/sdkforjava/?referer=');">download the SDK</a>, unzip it in the location of your choice, and then change the script to reflect your configuration.</p>
<p>If you are running R in Ubuntu, you should install rJava using apt-get instead of using install.packages() from inside of R:</p>
<blockquote><p>sudo apt-get install r-cran-rjava</p></blockquote>
<p>Here&#8217;s the codez. And a <a href="https://gist.github.com/722230" onclick="pageTracker._trackPageview('/outgoing/gist.github.com/722230?referer=');">direct link</a> for you guys reading this through an RSS reader:<br />
<script src="http://gist.github.com/722230.js"></script></p>
<p>I realize that Duncan Temple Lang has created the <a href="http://www.omegahat.org/RAmazonS3/" onclick="pageTracker._trackPageview('/outgoing/www.omegahat.org/RAmazonS3/?referer=');">RAmazonS3</a> package which can easily do what the above code sample does. The advantage of using rJava and the AWS Java SDK is the ability to apply the same approach to <strong>ALL</strong> the AWS services. And since Amazon maintains the SDK this guarantees that future AWS services and features will be supported as well.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.cerebralmastication.com/2010/11/controlling-amazon-web-services-using-rjava-and-the-aws-java-sdk/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Starting an EC2 Machine Then Setting Up a Socks Proxy&#8230; From R!</title>
		<link>http://www.cerebralmastication.com/2010/07/starting-an-ec2-machine-then-setting-up-a-socks-proxy-from-r/</link>
		<comments>http://www.cerebralmastication.com/2010/07/starting-an-ec2-machine-then-setting-up-a-socks-proxy-from-r/#comments</comments>
		<pubDate>Fri, 16 Jul 2010 22:07:12 +0000</pubDate>
		<dc:creator>JD Long</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[aws]]></category>
		<category><![CDATA[ec2]]></category>
		<category><![CDATA[proxy]]></category>
		<category><![CDATA[R]]></category>

		<guid isPermaLink="false">http://www.cerebralmastication.com/?p=748</guid>
		<description><![CDATA[I do some work from home, some work from an office in Chicago and some work on the road. It&#8217;s not uncommon for me to want to tunnel all my web traffic through a VPN tunnel. In one of my previous blog posts I alluded to using Amazon EC2 as a way to get around [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.cerebralmastication.com/wp-content/uploads/2010/07/firewallkat.jpg"><img class="alignleft size-full wp-image-765" title="firewallkat" src="http://www.cerebralmastication.com/wp-content/uploads/2010/07/firewallkat.jpg" alt="" width="361" height="312" /></a>I do some work from home, some work from an office in Chicago and some work on the road. It&#8217;s not uncommon for me to want to tunnel all my web traffic through a VPN tunnel. In one of my previous blog posts I <a href="http://www.cerebralmastication.com/2009/11/using-amazon-ec2-to-thwart-crappy-internal-it-services/">alluded to using Amazon EC2 as a way to get around your corporate IT</a> <span style="text-decoration: line-through;">mind control voyeurs</span> service providers. This tunneling method is one of the 5 or so ways I have used EC2 to set up a tunnel. I used to fire these tunnels up manually using the <a href="https://console.aws.amazon.com" onclick="pageTracker._trackPageview('/outgoing/console.aws.amazon.com?referer=');">Amazon AWS Management Console</a> then opening a shell prompt and entering:</p>
<blockquote>
<pre>ssh -i ~/MyPersonalKey.pem -D 9999 root@ec2-184-73-41-72.compute-1.amazonaws.com</pre>
</blockquote>
<p>the -i switch tells ssh to use my RSA identity file stored in ~/MyPersonalKey.pem</p>
<p>the machine name (ec2-184-73-41-72.compute-1.amazonaws.com) I get from the AWS Management Console</p>
<p>the -D is the magic. -D opens an dynamic port forwarding tunnel between my Linux box and the EC2 machine. This is, for all intent and purposes, an encrypted SOCKS4 proxy on port 9999 of localhost. Then I just have to change my proxy settings in Firefox to use use a SOCKS host.</p>
<p>Now that&#8217;s all pretty easy. And I like easy. But it&#8217;s not easy ENOUGH. You see, I&#8217;m lazy. I&#8217;m not just lazy in the &#8220;I&#8217;ll do it mañana&#8221; sort of way, but in the &#8220;I&#8217;m too damn lazy to click my mouse 5 times&#8221; way.</p>
<p>So I want this easier. Well, I can make the proxy settings in Firefox easier through the use of the <a href="https://addons.mozilla.org/en-US/firefox/addon/1557/" onclick="pageTracker._trackPageview('/outgoing/addons.mozilla.org/en-US/firefox/addon/1557/?referer=');">Quick Proxy extension for Firefox</a>. That&#8217;s a good start. It turns on and off the proxy with a single mouse click. But I still have to go into the AWS management web site, fire up a machine then log in via SSH. Let&#8217;s make that part easier!</p>
<p>While it&#8217;s not simple to install and configure, the EC2 command line tools are going to be required in order to make a script that fires up EC2 and then connects to the instance with ssh. I struggled getting the tools to run until I found <a href="http://linuxsysadminblog.com/2009/06/howto-get-started-with-amazon-ec2-api-tools/" onclick="pageTracker._trackPageview('/outgoing/linuxsysadminblog.com/2009/06/howto-get-started-with-amazon-ec2-api-tools/?referer=');">this tutorial</a>.</p>
<p>Your file locations and names may be different than the tutorial. Change appropriately. I followed the tutorial instructions but I created a key named ec2ApiTools which will come in handy later.</p>
<p>After you get the EC2 tool up and running and you can do something like list the available AMIs without an error you can stop with the tutorial. I&#8217;ve been doing a lot of shell scripting lately so I said to myself, &#8220;Self, let&#8217;s script the ssh connection in R!&#8221; For the record, I always end my impredicative in an explanation point which I verbally pronounce as, &#8220;BANG!&#8221; As a result, when I talk to myself it sounds like two 10 year old boys playing cops and robbers. Anyhow, I did script it with R using Rscript. Because I&#8217;m a man who listens to myself.</p>
<p>And since you were kind enough to slog through my channeling the drunken ghost of James Joyce, here&#8217;s my script:</p>
<script src="http://gist.github.com/478930.js"></script>
<p>If you&#8217;re reading this in an RSS reader of for some other reason don&#8217;t see an R script above, <a href="http://gist.github.com/478930#file_start_ec2_instance_ssh.r" onclick="pageTracker._trackPageview('/outgoing/gist.github.com/478930_file_start_ec2_instance_ssh.r?referer=');">here&#8217;s your link</a>.</p>
<p>The only two EC2 API commands I use in the script are  <em>ec2-run-instances</em> which starts the instance and <em>ec2-describe-instances</em> which gives me a list of running instances and their details.The rest of the script is simply parsing the output and figuring out which instances was started last.</p>
<p>I&#8217;ve now set up a launcher panel item that starts the script. Then when I see the xterm window come up I click the little red button in the lower right corner of my browser which switches on the Firefox proxy. Then I&#8217;m safe to surf <a href="http://www.sofmag.com/" onclick="pageTracker._trackPageview('/outgoing/www.sofmag.com/?referer=');">Soldier of Fortune Magazine</a> without the interference of my corp firewall.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.cerebralmastication.com/2010/07/starting-an-ec2-machine-then-setting-up-a-socks-proxy-from-r/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
	</channel>
</rss>

