Suddenly, Nashville

What I didn’t realize about Nashville is that it would be this cold. Seriously, it’s like Boston in March out there (of course, minus the snow). Hot chicken is planned for later in the week, work schedule permitting (ideally, I’d like to go to Hattie B’s this week, and then Prince’s next week, but we shall see).

I’m pausing now to start an amazing new dimension to my travelling life. After over three years of consulting at Mammoth Data, this time, this time, my friends, I have brought tea bags. I will be making a proper cup of tea in my hotel room for the first time.

(yes, the first time it occurred to me in all those years was last night around 11pm. And yes, I did treat it as if it was up there with discovering fire or gravity)

Back now. The lack of a spoon almost led me into stormy waters, but I discovered a plastic straw thing that seemed to work (it did not melt in the tea, so that’s a start). I feel that I need to bring a travel mug to make it a proper cup, though. And that means I can buy a new mug! So this new approach of high living is turning out dividends already.

First restaurant visit of the trip was Local Taco, which sadly made me think that it has been too long since I’ve been to La Vaquita. Not terrible, but not exciting either. Tomorrow, though, a Korean flavour: Soy Bistro. I’m hoping it’s good, because if it is, I’ll likely be eating there several times this week…

Datasets in Spark

The new dataset API lands today in Spark 1.6. Though it’s marked as experimental, it is soon to become an important part of the Spark ecosystem as the platform attempts to push the underlying RDD operations away from the gaze of the everyday developer and into the hands of the Catalyst optimization engine.

Datasets simply present a typed interface over Dataframes, providing a RDD-like way of interacting with the Dataframe with negligible overhead. Creating a dataset is simple:

val boardgameDataframe ="boardgames.json")
case class Boardgame(name: String, rating: Int, type: String)

val  boardgameDataset = boardgameDataframe[Boardgame]

Standard RDD-like operations and aggregations like map, filter, and groupBy are supported. If we want to filter the data so as to only report games rated 8 or above and then group them by type, we’d use the fairly familiar chain below:

val highRanking = boardgameDataset.filter(_.rating >= 8).groupBy(_.type)

A good question at this point is ‘why is this useful, and what advantage does it get me over just doing RDD or dataframe operations?’

Firstly, having a typed interface over dataframes gives you the chance of compile-time type-checking instead of discovering at run-time that the column you’re referencing either doesn’t exist, or is a Long instead of a String (of, course, I’m sure you’re doing testing to make sure this doesn’t happen in production, aren’t you?).

But more importantly, the type information that you provide to a dataset is used to automatically provide an encoder for the Tungsten memory management system (in a similar, but improved manner like Java or Kryo serialization), immediately offering memory benefits over RDDs. As the datasets are based on Dataframes, the execution of your operations is handled by Catalyst, so you’ll benefit from the query optimization engine.

Also, since Catalyst is going to be getting adaptive query optimization (follow SPARK-9850 for updates on this feature, which will allow Spark to dynamically update query plans depending on runtime information), getting used to working with datasets now is going to pay off in your Spark applications in the very near future.

Unitaskers - Accept No Substitutes

Yes, it looks like it has been buried in the dark corner of an Ohio warehouse for forty years.

A photo posted by Ian Pointer (@carsondial) on

But, come on, you can’t argue with the results. And they do get better as the days go on!


A photo posted by Ian Pointer (@carsondial) on

As you can imagine from the photo, they were pretty amazing. I got the recipe from Sprinklebakes, and used a butter / golden syrup / vanilla / yet more sugar mixture for the filling. Beware: looking at 250g of melted butter may induce diabetes. Or at least shame. Try to put those feelings past you and you’ll be rewarded with stroopwafel goodness. And a sugar coma. But it’s a small price to pay.

Since coming back to Durham, I’ve been taking advantage of finally being able to stock up my fridge for the first time since October. This naturally led to cooking all the things, including the aforementioned stroopwafels, a full roast dinner (complete with goose fat potatoes, Yorkshire puddings, and sous-vide beef, just to be completely crazy), apple crumble, pistachio butter, gianduja, and microwave cakes, a white chocolate namelaka…and brown butter ice-cream. Oh, and wontons with spicy chili oil. And a jalfrezi curry with the beef leftovers.

My concern is that I might have burnt myself out.

(only kidding; the Wonder Grinder is a revelation for nut butters - in less than 30 minutes, I had a pistachio butter smoother than anything the Vitamix could ever give me1)

Also, due to a last-minute cancellation, I had my first Escape Room experience this weekend! We escaped the 90s. I complained about the dates for the Spice Girls concert being all wrong and discovered that directional locks are A Thing. There may be more Escape Room antics in the future; apparently there’s one in Columbia too…

Now, though, the long Christmas break is over. I’ve been very lucky that I managed to wrangle a UK-style holiday over the past two weeks, but tomorrow, the year starts anew. Who knows where it’ll take me?2

  1. I am sorry, Vitamix. In this one instance, you failed me. [return]
  2. All accounts indicate Tennessee, but that’s as much as I know at the moment. [return]

Hello 2016

A brand new year deserves a new look. I’ve been meaning to make the site a little more friendly at different resolutions, but I just haven’t had the time. Instead, I took the advantage of the Christmas break to move from Jekyll to Hugo and bashed the Hyde-X theme about somewhat until I was mostly happy.

I’ve moved away from Hoefler Text and back to a sans-serif typeface Open Sans. Other than that, not much has changed; everything is still in Markdown, but oh my does the site rebuild faster in Hugo over Jekyll.

Plans for this year? Yes.

Melted Candles

We’ve reached the tipping point of the holiday where Christmas is over and the flight home is rapidly approaching. I even have to work for a bit tomorrow. Just enough time left for a brief last shop and to melt down the last of the Christmas candles.

A good Christmas, spent like many before, and hopefully many after.

(though, let’s hope fewer computer losses in the upcoming years)

Suddenly, London

It is, unfortunately, easy to get nostalgic for a place that maybe never existed, and if it did, was over as you were being born. But you can feel it as the hills roll past on the way to Haddenham and Thame Parkway, a tiny railway station on the way to Marylebone, traveling past enclosed fields that have likely had their enclosures set for hundreds of years. The seductive call of a Jerusalem that never was. Sheep dogs being trained on the hills

You come back once a year and you can feel the country changing. The creeping Americanisms, your home town moving on and building itself anew for the 21st century, ripping old parts out with no thought as to the memories you made in them. After all, you’ve gone now. It’s not your place anymore.

And yet, the muscle memory of knowing exactly when to turn, of the cottages racing by, replaced by the terraces; the post-war consensus survivors smiling at you and whispering ‘it wasn’t a dream, we did this, once’. The modern shoebox estates, where everything is crammed together and the green disappears. The towers as London approaches, much fewer in number, but Trellick still standing proud, brutal, and at long last, loved.

Walking up the steps to the Royal Festival Hall, Saint Etienne’s ‘London Belongs To Me’ coming on random play, like the iPhone just knows, big smile as the Queen Elizabeth Hall and Hayward Gallery come into view and the book sellers are putting out their tables. And walking up straight into a group of Pelican books. They knew, too.

But even as the Tate Modern’s huge chimney breaks out in front of you, there’s the realization - this place meant and means so much to you, and you’re happy to be back, but it’s not yours anymore. You don’t belong here in in the same way you did even five years ago. You can visit, but London will never belong to you. The construction cranes lining the sky across the water announce that it will likely not belong to your friends here for that long, either.

Saying all that, had a good time in London yesterday.

Suddenly, Britain


It takes a bit of getting used to being back in a bedroom that is literally smaller than my bathroom, and I’m not entirely convinced that the sun even came up today, but it’s good to be back home.

I’m not entirely sure that my sister feels exactly the same way, seeing as how I’ve spent the past 48 hours wearing a Father Christmas hat and making sure that she’s full of festive cheer. I have to hide the knives1.

A week of work awaits, then a quick trip to London (my first in two years!!), and finally: CHRISTMAS!

plays Slade very loudly

  1. though she probably already scored the biggest burn of the holiday by saying that listening to my music is like being trapped in Phonogram. [return]

Suddenly, Durham

The week in Atlanta was…odd. I spent most of it in the hotel as the client’s office firewall prevented me from talking to things on the Internet I needed to use to do my job. On the bright side, it meant that I was able to talk everybody into my staying here in Durham this week (given that I’ve been essentially working remote this week anyhow). So, hurrah! A full, final week in Durham before I fly home for Christmas. I just have to finish packing. is going to be down this year - I simply didn’t have to get it all reactivated in time (or add the new bits I wanted). There will hopefully be something to ring in the new year, though, if I can get it finished on time…

Suddenly, Atlanta

China didn’t happen, so instead I found myself helping to make a Thanksgiving dinner in South Carolina a large portion of last week. And a good time was had by all; Nerf wars that began at 7am (you have to get up early to fire foam darts at your brother, it seems), three different board games, pull-apart stuffing rolls that we didn’t burn to within an inch of their lives (thus atoning for last year’s disaster), and also answering the question ‘what do you do with a kilo of pound cake that didn’t come out of the pan properly?’

(pound cake pudding, it turns out)

Then, all of a sudden, back in the car, driving back to NC early in order to prepare for a last-minute trip to Atlanta. A last-minute trip that will run pretty much up to December 11th, when I fly home to the UK. It’s not exactly the gentle ending of the year I was expecting.

smartctl says no

It has been probably about a month, but I’m still trying not to think about the mess of computer parts on my bedroom floor. In particular, a dead 1Tb drive that contains over ten years of accumulated data.


And of course, like a proper person involved with computers, I tell people to make backups all the time, but is there a backup for this drive? Ha. Funny story: I was upgrading the firmware on my NAS to enable S3 backup when the HDD died. There may have been cosmic levels of rage when I discovered it was dead, but I could only maintain that for five or ten minutes because I still had work to do. Now that I’m in something of a fallow period (the trip to China isn’t going to happen at this point), I have time to come back to it.

Initial attempts to resuscitate the drive have not gone well; I suspect that I may have already wrecked things with a wild fsck. I forgot the first rule - make a disk image if you can. Right now, I’ve put it through the chamber vacuum sealer and thrown it in the freezer for the last attempt to save it tomorrow before I give it up for good.

The worst thing is that I really don’t know the extent of what I’ve lost. I know that there were rare films, TV shows, and comics on there, which is annoying but not entirely irreplaceable (the complete works of Grant Morrison! Including Near Myths and Steed & Mrs. Peel!). But I had also used it as a place to store things like the Quartz Composer installations I’d made to relieve space on my newer SSD setups. Some of that may be recoverable from my Time Machine backups. If I’m lucky.

It’s not as bad as the black mold in the basement on Trinity. But I can’t help the feeling of loss, even if it is just a bunch of zeros and ones on an ext3 filesystem.

But, like a good British person, I will go back to repressing it and lining up the torrents to rebuild.