Shell scripting EC2 for fun and profit

Lately I’ve been doing some work with creating ad-hoc clusters of EC2 machines. My ultimate goal is to create a simple way to spin up a cluster of EC2 machines for use with Bryan Lewis’s very cool doRedis backend for the R foreach package. But that’s a whole other post. What I was scratching my head about today was that I’d really just like to, with a single command, spin up an EC2 instance, wait for it to come up, and then ssh into it. I do this iteration about 20 times a day when I’m testing things, so it seemed to make sense to shell script it.
To do this, one needs the EC2 command line tools installed on your workstation. In Ubuntu that’s as easy as `sudo apt-get ec2-api-tools`

So here’s a short shell script to spin up an instance, wait 30 seconds, then connect:

If you’re reading this through an RSS reader, you can see the script over at github.

Obviously you’ll need to change the parameters at the top of the script to suit your needs. But since this was a bit of a pain in the donkey hole for me to figure out, I thought I would share.

If you want to help out, I’d love you to enlighten me on how to have the script figure out if an instance has finished booting so I could eliminate the sleep step.

10 Comments

  1. ken says:

    have the instance open a port (eg a via netcat or a webport) and try connecting in a loop, or just ssh with a command instead of interactive and check the status in a loop.

  2. Matt Pettis says:

    Very nice! I will be stealing this…

  3. Matt Pettis says:

    Very nice! I will be stealing (and, may have posted this already, browser hung on submit…)

  4. If you can modify the AMP as well, then you can something simple such as having /etc/rc.local have a command ‘date > /var/www/DoneBooting.txt’. Your shell script could then loop, test for said file via wget or curl, sleep for a second or two and try again. Or you leet the AMI instance tweet to a s3cr33t account and monitor that. Just kidding….

  5. Paul says:

    http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html

    There is a status attribute you can check, this still means polling the instance with ec2-describe-instances.

    As suggested, you could run a script after the instance starts which actually ping you instead.
    Many people launch a skeleton instance and use that “after start” functionality to load software, scripts and data. If you combine with a provisioning tool such as puppet/chef, you have the instance ask the puppet/chef master what it needs to install.

  6. John Dennison says:

    Brilliant i have been struggling of how to capture the newly created instance id so i could pass it to my sftp/ssh. this is exactly that. one tweak to the ssh command. I’m running on fedora and my key as set to the standard system variable. I had to change the order of the RSA command else the ssh would fail. ie it needed to add the key my list of trusted servers. if i used:

    ssh -o StrictHostKeyChecking=no ubuntu@$name

    it works!!

  7. Siah says:

    Nice post.

    Can you ping your instance? do they get some sort of sub domain name or IP? if yes you can then ping it with regular ping and continue as soon as it responds.

  8. Mike says:

    Have you checked out StarCluster, out of MIT? That sounds like it might already do almost all of what you’re looking to do….

    http://web.mit.edu/stardev/cluster/

  9. John Dennison says:

    I have been working to automate the creation of ec2 for automated data runs. I have stolen your script and added some redundancies. I was having alot of problems with the EC2 machine denying my initial ssh requests so i built in if statement to retry. I am using a keypair already on the AMI. so no need for amazon key.

    instance_id=$(ec2-run-instances -g $secGroup -t t1.micro $ami | awk ‘/INSTANCE/{print $2}’)
    echo $instance_id

    name=$(ec2-describe-instances $instance_id | awk ‘/INSTANCE/{print $4}’)
    while [ $name = pending ] ; do
    sleep 5
    name=$(ec2-describe-instances $instance_id | awk ‘/INSTANCE/{print $4}’)
    echo $name
    done
    sleep 10
    OUT=0
    OUT1=0

    ssh -t -t -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no jdennison@$name< exit
    EOF
    OUT=$?
    if [ $OUT -ne 0 ];
    then
    echo “failed”
    sleep 15
    ssh -t -t -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no jdennison@$name< exit
    EOF1
    OUT1=$?
    fi

    if [ $OUT1 -ne 0 ];
    then
    echo “SECOND FAIL”
    sleep 10
    fi

  10. John Dennison says:

    The EOF1 redirects got deleted in the copy. they should look like

    ssh -t -t -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no jdennison@$name<<EOF1
    exit
    EOF1

Leave a Reply