Things have been a bit busy. Firstly, this happened yesterday:

Yes, I now own a car. It surprises me almost as much as it does you. As of yesterday, I have finally given up the long experiment of getting around the Triangle by public transport. It worked out better than most would expect, but I am a little tired of having two hour round-trips to Whole Foods.

I will, of course, still walk to work. Because I’m strange.

When I last wrote, I was in Hawaii. This week, my family have been here in NC instead. Their week here has mostly consisted of eating, meeting my friends, and putting up shelves. Oh, and Mum spent most of the day today cooking and cursing American ovens and pans.

Will be sad to see them go tomorrow, but at least in this day and age, we have FaceTime.

On Friday, I was told by a ten year old boy that my bedroom is what he’d want when he grows up. I’m not sure exactly if that’s a revelation about my stunted emotional growth or just that I have a lot of cool things. Despite that, I still didn’t let him transform Jetfire. Flattery only goes so far, people.

Anyway, two days of travel later, I’m in Hawaii. Which is weird, but I’m grateful that they eased me in via an airport that is still emblazoned with 1970s-era Eurostile and Helvetica, plus concrete walls as far as the eye can see. Bizarrely, it reminded me a lot of LAX circa 1994, on my first-ever trip to America (yes, it’s been twenty years. I am old). LAX, on the other hand, was selling macarons and being all fancy.

So far, we have driven along a road for a while, had some Korean BBQ, taken the family on a supermarket visit, bought spam, accidentally found a hipster burger place, and almost got swept out into the ocean while trying to rescue the flip-flops that I had bought less than an hour earlier. Eventful, and we still haven’t gone very far yet…

One of my favourite new technologies of the moment is Ansible. It’s the new kid on the block alongside stalwarts like Puppet or Chef, but it wants to do more with less. Both Puppet or Chef are great for configuration management, but take some getting used to if you’re a sysadmin coming across them for the first time. In addition, they require extra servers, extra overhead on the servers that they maintain configuration, and you still might find yourself resorting to cluster ssh in order to send a command to a bunch of servers there and then. In addition, you’ll often find Puppet or Chef shops using a different tool for application deployment (Capistrano for Rails, Fabric for Django). Ansible is a step back in some ways, but steps forward in others. It sidesteps the declarative model of Puppet, and abandons the idea of running agents on servers; instead, everything runs over standard SSH (and for those of you sceptical that SSH can scale on that front, Rackspace is currently using Ansible across tens of thousands of virtual machines). In addition, workflows like orchestration, deployment, and ad-hoc commands to groups of servers are all present and fully supported. It’s a great tool for the devops basket.

Ansible is still pretty new (it’s just about to celebrate its second birthday), but is coming along at a fast and furious pace and is ready for production right now. Indeed, development is so active that many simply track the GitHub repo rather than waiting for the point releases. One of the features I’ve been looking froward to landing is the ability to create spot instances in the Amazon cloud. It was merged into the main branch just under two weeks ago and while it’ll be available in Ansible 1.6, it’s ready for use today with a handy git pull.

Here’s a stripped-down playbook (the equivalent of a Puppet manifest or Chef recipe) that launches a number of spot instances at a specified price (spot_count and spot_price are passed in on the command-line using the extra-vars argument).

---
- name: Spin up spot instances
  hosts: 127.0.0.1
  connection: local
  tasks:
    - name: create  {{ spot_count }} spot instances with spot_price of ${{ spot_price }}      
      local_action:
        module: ec2
        region: us-west-2
        spot_price:  '{{ spot_price }}'
        spot_wait_timeout: 180
        keypair: example-keypair
        instance_type: t1.micro
        image: ami-ccf297fc
        wait: yes
        group: test-group
        count:  '{{ spot_count }}'
      register: ec2

    - name: Tag instances
      local_action: ec2_tag resource= region=us-west-2 state=present
      with_items: ec2.instances
      args:
        tags:
          Spot: '{{ spot_price }}'

As you can see, it’s pretty straightforward. The hosts/connections preamble makes sure that the following commands run on my local box instead of a remote machine, and then we get into the list of tasks that need to be performed (Ansible takes the approach that tasks are run in the order they’re specified. Which seems obvious, but if you’re coming in from the Puppet/Chef world, you may have already just cried a huge sigh of relief).

Anyway, onto the tasks themselves. The first creates the spot requests. Ansible takes a ‘batteries included’ approach, supplying a boatload of modules that do everything from running shell commands to altering hardware routers. Here, we’re using the ec2 module to talk to Amazon Web Services. The options passed into the module should make sense if you’re familiar with the AWS setup; we need to specify a region where our instances will live (Oregon/us-west-2 in this example), our instance_type (we’re being cheap and using micro instances), security groups, AMI image (just using the stock Amazon Linux AMI) and the SSH keypair. We also set our suggested spot price and how many instances we would like, and we do that by using Ansible’s templating functions; the ‘’ sections ensure that our variables that we specify on the command-line will be filled into the right place at run-time. We also need a waiting period to be set, as AWS won’t fulfill the request instantly, so we’re waiting three minutes to get our machines or else we’ll terminate.

Assuming that our spot request bid succeeds, the ec2 module will return information back to Ansible about the instances that have been created. We capture that information with the register: ec2 line, and then use it in our next task, which tags the newly-created instances with the spot price we used to create them (as you can imagine, the chaining of output from the previous task into further tasks is a very useful feature that reoccurs throughout Ansible’s design).

And here’s the output from running this play:

$ ansible-playbook spot.yml -i spot.ini --extra-vars "spot_price=0.005 spot_count=2" 

PLAY [Spin up spot instances] ************************************************* 

GATHERING FACTS *************************************************************** 
ok: [127.0.0.1]

TASK: [create 2 spot instances with spot_price of $0.005] ********************* 
changed: [127.0.0.1]

TASK: [Tag instances] ********************************************************* 
changed: [127.0.0.1] => (item={u'kernel': u'aki-fc8f11cc', u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-11-32.us-west-2.compute.internal', u'public_ip': u'54.186.184.201', u'private_ip': u'172.31.11.32', u'id': u'i-f51760fd', u'state': u'running', u'virtualization_type': u'paravirtual', u'architecture': u'x86_64', u'ramdisk': None, u'key_name': u'housepi', u'image_id': u'ami-ccf297fc', u'public_dns_name': u'ec2-54-186-184-201.us-west-2.compute.amazonaws.com', u'state_code': 16, u'placement': u'us-west-2c', u'ami_launch_index': u'0', u'dns_name': u'ec2-54-186-184-201.us-west-2.compute.amazonaws.com', u'region': u'us-west-2', u'launch_time': u'2014-03-23T18:44:12.000Z', u'instance_type': u't1.micro', u'root_device_name': u'/dev/sda1', u'hypervisor': u'xen'})
changed: [127.0.0.1] => (item={u'kernel': u'aki-fc8f11cc', u'root_device_type': u'ebs', u'private_dns_name': u'ip-172-31-14-189.us-west-2.compute.internal', u'public_ip': u'54.186.112.206', u'private_ip': u'172.31.14.189', u'id': u'i-bb1661b3', u'state': u'running', u'virtualization_type': u'paravirtual', u'architecture': u'x86_64', u'ramdisk': None, u'key_name': u'housepi', u'image_id': u'ami-ccf297fc', u'public_dns_name': u'ec2-54-186-112-206.us-west-2.compute.amazonaws.com', u'state_code': 16, u'placement': u'us-west-2c', u'ami_launch_index': u'0', u'dns_name': u'ec2-54-186-112-206.us-west-2.compute.amazonaws.com', u'region': u'us-west-2', u'launch_time': u'2014-03-23T18:44:12.000Z', u'instance_type': u't1.micro', u'root_device_name': u'/dev/sda1', u'hypervisor': u'xen'})

PLAY RECAP ******************************************************************** 
127.0.0.1                  : ok=3    changed=2    unreachable=0    failed=0 

If I look at my EC2 console afterwards, these instances are up and running, plus they have the spot price tagged as requested. From here, the instances can be provisioned further; installing databases, webservers, and so on; adding them to load balancers, updating DNS entries - pretty much anything you would do manually can be automated with Ansible in a sensible manner. With no agents or fiddly PKI management. Hurrah!

Some other great things about Ansible: firstly, they’re very welcoming towards new contributors (indeed, the play I described above uncovered a small bug in the ec2 module in regards to how it handled instance counts - I fixed the bug, made a pull request, and it was merged into master before the day was out). And, for all you hipsters out there, the Ansible company is located in downtown Durham. So it’s not just an automation system, it’s a local automation system.

I am but a few discs away from completing my once-every-couple-of-years rewatching Gilmore Girls. It’s at the point where I just want to throw rocks at Christopher and Logan whilst scratching my head at the personality transplant they gave Marty. Oh, Season 7, you were such a mistake. But at least you ended properly.

Next week, I’ll be heading back to RDU once again. But this time, heading back to LA, not for Santa Monica-related shenanigans, but just as a stopover on the way for a week in Hawaii with the family!

A tip for those of you using EventMachine in Ruby. As you may know, by default EventMachine uses the select() system call during its run through the event loop to check for new inputs on file descriptors (which it then uses to hand off to callbacks you may have registered). This is useful, as you’ll find select() on pretty much any UNIX-based system you can care to name. However, there are a few drawbacks:

• select() is often limited to FD_SETSIZE file handles, which is normally 1024.
• Because of the way _select()_ runs, EventMachine needs to loop through all the file descriptors twice, once for passing into select(), and once again to see if it has marked a descriptor ready for reading or writing.

This is often fine during developing, and perhaps even testing (especially if you’re not load-testing properly), but you may be in for a surprise when you start getting serious traffic in production.

Thankfully, EventMachine comes with a few strategies to get around this issue. Sadly, it’s not quite as simple as select(), but not too taxing. Both Linux and BSD-derived systems have taken different approaches - Linux provides a system called epoll, while BSD systems have kqueue. Both implementations eliminate the idea of having to read through all file descriptors twice on every call, and are easier to scale past the 1024-descriptor limit. All you have to do is call EM.epoll or EM.kqueue before you start the Reactor.

Having said that, you’ll see a lot of code around the net that looks a bit like this:

EM.run do 
	EM.epoll
	...
	...
end

There’s a slight problem here. It should look like this:

EM.epoll
EM.run do 
	...
	...
end

There’s not much difference there, but the ordering of the lines is incredibly important - because EM.epoll in the first example is inside the Reactor, it will do nothing and will instead fall back to select(). And your code will blow up when it hits a big traffic spike. So be careful out there with rogue function snippets, and always set up with kqueue or epoll before you make the fateful call to EventMachine.run.