Been a while...

Things have been a bit busy. Firstly, this happened yesterday:

me and a car!' width=480

Yes, I now own a car. It surprises me almost as much as it does you. As of yesterday, I have finally given up the long experiment of getting around the Triangle by public transport. It worked out better than most would expect, but I am a little tired of having two hour round-trips to Whole Foods.

I will, of course, still walk to work. Because I’m strange.

When I last wrote, I was in Hawaii. This week, my family have been here in NC instead. Their week here has mostly consisted of eating, meeting my friends, and putting up shelves. Oh, and Mum spent most of the day today cooking and cursing American ovens and pans.

Will be sad to see them go tomorrow, but at least in this day and age, we have FaceTime.


On Friday, I was told by a ten year old boy that my bedroom is what he’d want when he grows up. I’m not sure exactly if that’s a revelation about my stunted emotional growth or just that I have a lot of cool things. Despite that, I still didn’t let him transform Jetfire. Flattery only goes so far, people.

Anyway, two days of travel later, I’m in Hawaii. Which is weird, but I’m grateful that they eased me in via an airport that is still emblazoned with 1970s-era Eurostile and Helvetica, plus concrete walls as far as the eye can see. Bizarrely, it reminded me a lot of LAX circa 1994, on my first-ever trip to America (yes, it’s been twenty years. I am old). LAX, on the other hand, was selling macarons and being all fancy.

So far, we have driven along a road for a while, had some Korean BBQ, taken the family on a supermarket visit, bought spam, accidentally found a hipster burger place, and almost got swept out into the ocean while trying to rescue the flip-flops that I had bought less than an hour earlier. Eventful, and we still haven’t gone very far yet…

Spot Instances With Ansible

One of my favourite new technologies of the moment is Ansible. It’s the new kid on the block alongside stalwarts like Puppet or Chef, but it wants to do more with less. Both Puppet or Chef are great for configuration management, but take some getting used to if you’re a sysadmin coming across them for the first time. In addition, they require extra servers, extra overhead on the servers that they maintain configuration, and you still might find yourself resorting to cluster ssh in order to send a command to a bunch of servers there and then. In addition, you’ll often find Puppet or Chef shops using a different tool for application deployment (Capistrano for Rails, Fabric for Django). Ansible is a step back in some ways, but steps forward in others. It sidesteps the declarative model of Puppet, and abandons the idea of running agents on servers; instead, everything runs over standard SSH (and for those of you sceptical that SSH can scale on that front, Rackspace is currently using Ansible across tens of thousands of virtual machines). In addition, workflows like orchestration, deployment, and ad-hoc commands to groups of servers are all present and fully supported. It’s a great tool for the devops basket.

Ansible is still pretty new (it’s just about to celebrate its second birthday), but is coming along at a fast and furious pace and is ready for production right now. Indeed, development is so active that many simply track the GitHub repo rather than waiting for the point releases. One of the features I’ve been looking froward to landing is the ability to create spot instances in the Amazon cloud. It was merged into the main branch just under two weeks ago and while it’ll be available in Ansible 1.6, it’s ready for use today with a handy git pull.

Here’s a stripped-down playbook (the equivalent of a Puppet manifest or Chef recipe) that launches a number of spot instances at a specified price (spot_count and spot_price are passed in on the command-line using the extra-vars argument).

- name: Spin up spot instances
  connection: local
    - name: create  {{ "{{ spot_count " }}}} spot instances with spot_price of ${{ "{{ spot_price " }}}}      
        module: ec2
        region: us-west-2
        spot_price:  '{{ "{{ spot_price " }}}}'
        spot_wait_timeout: 180
        keypair: example-keypair
        instance_type: t1.micro
        image: ami-ccf297fc
        wait: yes
        group: test-group
        count:  '{{ "{{ spot_count " }}}}'
      register: ec2

    - name: Tag instances
      local_action: ec2_tag resource={{ }} region=us-west-2 state=present
      with_items: ec2.instances
          Spot: '{{ "{{ spot_price " }}}}'

As you can see, it’s pretty straightforward. The hosts/connections preamble makes sure that the following commands run on my local box instead of a remote machine, and then we get into the list of tasks that need to be performed (Ansible takes the approach that tasks are run in the order they’re specified. Which seems obvious, but if you’re coming in from the Puppet/Chef world, you may have already just cried a huge sigh of relief).

Anyway, onto the tasks themselves. The first creates the spot requests. Ansible takes a ‘batteries included’ approach, supplying a boatload of modules that do everything from running shell commands to altering hardware routers. Here, we’re using the ec2 module to talk to Amazon Web Services. The options passed into the module should make sense if you’re familiar with the AWS setup; we need to specify a region where our instances will live (Oregon/us-west-2 in this example), our instance_type (we’re being cheap and using micro instances), security groups, AMI image (just using the stock Amazon Linux AMI) and the SSH keypair. We also set our suggested spot price and how many instances we would like, and we do that by using Ansible’s templating functions; the ‘{{ }}’ sections ensure that our variables that we specify on the command-line will be filled into the right place at run-time. We also need a waiting period to be set, as AWS won’t fulfill the request instantly, so we’re waiting three minutes to get our machines or else we’ll terminate.

Assuming that our spot request bid succeeds, the ec2 module will return information back to Ansible about the instances that have been created. We capture that information with the register: ec2 line, and then use it in our next task, which tags the newly-created instances with the spot price we used to create them (as you can imagine, the chaining of output from the previous task into further tasks is a very useful feature that reoccurs throughout Ansible’s design).

And here’s the output from running this play:

$ ansible-playbook spot.yml -i spot.ini --extra-vars "spot_price=0.005 spot_count=2" 

PLAY [Spin up spot instances] ************************************************* 

GATHERING FACTS *************************************************************** 
ok: []

TASK: [create 2 spot instances with spot_price of $0.005] ********************* 
changed: []

TASK: [Tag instances] ********************************************************* 
changed: [] => (item={u'kernel': u'aki-fc8f11cc', u'root_device_type': u'ebs', u'private_dns_name': u'', u'public_ip': u'', u'private_ip': u'', u'id': u'i-f51760fd', u'state': u'running', u'virtualization_type': u'paravirtual', u'architecture': u'x86_64', u'ramdisk': None, u'key_name': u'housepi', u'image_id': u'ami-ccf297fc', u'public_dns_name': u'', u'state_code': 16, u'placement': u'us-west-2c', u'ami_launch_index': u'0', u'dns_name': u'', u'region': u'us-west-2', u'launch_time': u'2014-03-23T18:44:12.000Z', u'instance_type': u't1.micro', u'root_device_name': u'/dev/sda1', u'hypervisor': u'xen'})
changed: [] => (item={u'kernel': u'aki-fc8f11cc', u'root_device_type': u'ebs', u'private_dns_name': u'', u'public_ip': u'', u'private_ip': u'', u'id': u'i-bb1661b3', u'state': u'running', u'virtualization_type': u'paravirtual', u'architecture': u'x86_64', u'ramdisk': None, u'key_name': u'housepi', u'image_id': u'ami-ccf297fc', u'public_dns_name': u'', u'state_code': 16, u'placement': u'us-west-2c', u'ami_launch_index': u'0', u'dns_name': u'', u'region': u'us-west-2', u'launch_time': u'2014-03-23T18:44:12.000Z', u'instance_type': u't1.micro', u'root_device_name': u'/dev/sda1', u'hypervisor': u'xen'})

PLAY RECAP ********************************************************************                  : ok=3    changed=2    unreachable=0    failed=0 

If I look at my EC2 console afterwards, these instances are up and running, plus they have the spot price tagged as requested. From here, the instances can be provisioned further; installing databases, webservers, and so on; adding them to load balancers, updating DNS entries - pretty much anything you would do manually can be automated with Ansible in a sensible manner. With no agents or fiddly PKI management. Hurrah!

Some other great things about Ansible: firstly, they’re very welcoming towards new contributors (indeed, the play I described above uncovered a small bug in the ec2 module in regards to how it handled instance counts - I fixed the bug, made a pull request, and it was merged into master before the day was out). And, for all you hipsters out there, the Ansible company is located in downtown Durham. So it’s not just an automation system, it’s a local automation system.

Hit Him Again, Luke

I am but a few discs away from completing my once-every-couple-of-years rewatching Gilmore Girls. It’s at the point where I just want to throw rocks at Christopher and Logan whilst scratching my head at the personality transplant they gave Marty. Oh, Season 7, you were such a mistake. But at least you ended properly.

Next week, I’ll be heading back to RDU once again. But this time, heading back to LA, not for Santa Monica-related shenanigans, but just as a stopover on the way for a week in Hawaii with the family!

It's 3am, Is Your EventMachine Selecting?

A tip for those of you using EventMachine in Ruby. As you may know, by default EventMachine uses the select() system call during its run through the event loop to check for new inputs on file descriptors (which it then uses to hand off to callbacks you may have registered). This is useful, as you’ll find select() on pretty much any UNIX-based system you can care to name. However, there are a few drawbacks:

  • select() is often limited to FD_SETSIZE file handles, which is normally 1024.
  • Because of the way _select()_ runs, EventMachine needs to loop through all the file descriptors twice, once for passing into select(), and once again to see if it has marked a descriptor ready for reading or writing.

This is often fine during developing, and perhaps even testing (especially if you’re not load-testing properly), but you may be in for a surprise when you start getting serious traffic in production.

Thankfully, EventMachine comes with a few strategies to get around this issue. Sadly, it’s not quite as simple as select(), but not too taxing. Both Linux and BSD-derived systems have taken different approaches - Linux provides a system called epoll, while BSD systems have kqueue. Both implementations eliminate the idea of having to read through all file descriptors twice on every call, and are easier to scale past the 1024-descriptor limit. All you have to do is call EM.epoll or EM.kqueue before you start the Reactor.

Having said that, you’ll see a lot of code around the net that looks a bit like this: do 

There’s a slight problem here. It should look like this:

EM.epoll do

There’s not much difference there, but the ordering of the lines is incredibly important - because EM.epoll in the first example is inside the Reactor, it will do nothing and will instead fall back to select(). And your code will blow up when it hits a big traffic spike. So be careful out there with rogue function snippets, and always set up with kqueue or epoll before you make the fateful call to

The Phantom Bedbugs

Most of this week was spent lying awake at night feeling all sorts of bugs crawling over me. These are the annoying phantom bugs that you somehow know are there, but can’t actually see or feel. And while they don’t exist, it actually doesn’t help at 2am when you turn on the light and rifle through the bedclothes trying to find any evidence of the things you just know are there.

It started when I saw the doctor on Tuesday. I have psoriasis, and before/during my trip to Boston, it seemed to be flaring up pretty badly, and even spreading to my legs, though spreading by drying out spot-like things instead of the normal scales like I find on the back of my head. I thought a trip to the doctor’s would be useful.

She ruled out spreading psoriasis pretty early on, and instead suggested that they could be bedbug bites. My knowledge of bedbugs is really just limited to seeing BBC reports on the infestations in places like New York, so I was somewhat horrified, as I had dim recollections of people having to throw out their mattresses, beds, clothes, and all sorts to other things to try and stem the relentless hordes of bugs.

Needless to say, I itched all the way back to work.

When I got back home, armed with Google Image search, I stripped the bed down to the mattress and went bug-hunting. And of course, I couldn’t find anything, but that didn’t mean they weren’t there. So sleep was somewhat harder Tuesday night.

As I couldn’t find anything, but I did want to sleep again sometime, I called Triangle Pest Control, who offer a free bedbug inspection, as well as impressive heat treatments where they heat your house up to 60˚C and KILL EVERYTHING IN SIGHT. The head of the company came around Friday afternoon, told me stories about their bedbug-hunting dogs, horror stories about a certain fast-food franchise in the Durham area, and then pulled up the mattress and the bed to confirm that he couldn’t find anything either.

So, it looks like it isn’t bedbugs. So, not entirely sure why my legs are the way that they are, but we can at least eliminate them. Hurrah!


It’s been a very tough week, for reasons that I can’t go into here right now. However, you should go see this, as it brightens up even a very dark week:


It has begun. Slowly, insidiously, and completely unexpectedly. And I was betrayed by Kinder Eggs, of all things. I have a pack sitting by my table which are waiting to be picked up by a friend. As I’ve had them since Christmas, I wanted to make sure that they were still good, so I flipped the box over and looked at the expiry date:


For a moment, I thought ‘oh no, they’re out of date’. For a moment. One second later, my brain reasserted itself and I realised it was saying September, not January. A small slip, but a worrying one. I think I over-compensated and refused to give out any temperatures in Fahrenheit for the rest of the day. And had three cups of tea. You can’t be too careful.

I accidentally walked 7.5 miles yesterday. It was sunny! And I hadn’t been about Durham for ages! The blisters I have on my big toe today were totally worth it! Well, maybe not that last one. I did a loop from my house down to Whole Foods and 9th Street and came back home via Trinity, Geer, and Gilbert. You can argue against the evils of gentrification until the cows go home and make tea for two (no beef, please), but it’s amazing how active Geer & Rigsbee is these days, not just with the bars and restaurants there, but the park just behind Cocoa Cinnamon, which almost always has children playing football at the weekend (and yes, I’m talking about real football here, so bonus points allocated), people milling about enjoying the sun, and the feeling that the pedestrian is safe to walk wherever (despite it actually being a road…I guess that could potentially be a problem!).

And then on the way back, I fell in love with another building. I can’t believe I’d never seen it before, but down on Gilbert is the former Hosiery Mills Dye House. From the outside, it looks amazing, a 1920s warehouse with great fronts and features. Once you look at the OpenDurham link, though, the inside is even better - look at the wooden trusses! LOOK AT THEM! OpenDurham says that the Durham Food Bank is currently situated there, but it looked empty yesterday, with a ‘for lease’ sign out front, and the Durham Food Bank website says that they’re moving. If anybody fancies a renovation on the lines of the Haw River Ballroom but in East Durham, they should come this way.

The Apples Were Quite Cold

I did have grand plans. I was going to go into Boston, visit all sorts of hipster places (there’s a tapas restaurant that is also a record and book shop, for crying out loud), take lots of pictures of my favourite building style, and generally enjoy my time in the area.

Except: the snow (and other, non-weather related concerns that meant that I didn’t feel much like going on big adventures during the night). Trying to venture through three feet or so of snow really wasn’t my idea of fun, so in the end I only made it into Boston on one day. A taxi ride to Alewife Station, and then the Red Line into Harvard Square, where it took me an embarrassing amount of time to rediscover Newbury Comics. Which seemed much smaller than I remember (admittedly, it is almost twenty years since I first went there pause for feeling old). There I wandered around trying to warm up, getting in people’s way, awkwardly responding to the conversation of the woman behind the counter, and looking at the glitter. So aside from the size, it was almost exactly like the time I went in 1997. Oho!

(I walked along thinking I wasn’t going to get anything. Then I saw the Criterion section and thought to myself “well, if they have a copy of Hard Boiled, I’ll get it, knowing that it’s fairly rare these days. And lo, they had one. And yes, it then went to the UK, then back to Boston, and only now back to Durham. A frequent flyer DVD)

I went to books, finding myself coming out of Park Street Station and knowing exactly where I was, looking at a CEX shop front that once used to be HMV in one direction, and Boston Common in the other. I walked past the giant Citgo sign as I walked from the centre of the city to Mei Mei. It was…further than I expected. And colder, despite the gloves and the hat and the scarf.

Finally, I did my Good Will Hunting bit, at the Au Bon Pain at Harvard Square. Except the snow covered all of the outside seating where the film scenes were shot, so I had tomato soup while watching two girls play Settlers of Catan.

It was an odd few weeks; I spent most of it trapped inside an office building, the (actually quite nice) Holiday Inn Express at Waltham, or on a plane somewhere. Hopefully when I go back in May I’ll have a chance for a little more exploring. And that the snow will have finally melted.

Alewife Station