San Francisco


A photo posted by Ian Pointer (@carsondial) on

Brownie flight.

A photo posted by Ian Pointer (@carsondial) on


A photo posted by Ian Pointer (@carsondial) on

Did I really just walk all that way? (Yes)

A photo posted by Ian Pointer (@carsondial) on


A photo posted by Ian Pointer (@carsondial) on

The legally-required photo of a tram in SF.

A photo posted by Ian Pointer (@carsondial) on

Om nom nom

A photo posted by Ian Pointer (@carsondial) on

Suddenly, Columbia???

I’m writing this from the second gate change in Charlotte airport waiting for a flight to San Francisco, haven been driven up by Tammy from Columbia. Now, my intended weekend was to relax after a rather stressful deployment to Nashville1 and then roll into work on Monday to finish off a project that has been hanging over me for…oooh, about three months.

The snowstorm put a pretty stop to that. Instead of flying home on a leisurely basis on Friday afternoon, I had to rush back to the airport in Nashville on Thursday night…but all the flights to RDU had already been cancelled, so instead I found myself flying to Columbia, SC, to spend a rather impromptu visit (and I think I caused sadness by not having brought any board games with me. So, a note: along with the passport, always carry an emergency board game). It was a fun weekend! Just not the one I was expecting.

And instead of gingerly making my way back to Durham today, it’s San Francisco. My first-ever visit to Silicon Valley. This has the potential to be exciting, but I can’t help but feel that it would have been better if it was next week instead of right now…

  1. ask me sometime about how I managed to avoid re-writing the Spark Thrift Server by writing a one-line Scala function that turns a Spark Dataframe into a Hive DDL string and some hasty Hive Metastore schema changes. It’s exciting! Or…not. [return]

An Infinite Tiling of People

After a bad week where it seemed like a Hadoop cluster generated every possible problem (up to and including spontaneously generating new cluster IDs for the namenode and datanodes), it was nice to be back in Durham, even only for a short time.

But, no time for love, Dr. Jones, as after waking up on Saturday morning, it was time to re-pressurize my back right tyre and head all the way to Raleigh where I met up with Tammy, Robert, the two smaller humans, and Christie for a visit to the Escher exhibit at the NC Museum of Art. All this time I’ve lived here, and this was the first time I’ve visited!

I can definitely recommend catching the exhibit in the upcoming (and last week), with one tiny caveat: if you don’t like crowds, try and go during the weekday if possible. There were so many people around that it did at times tip over into being ‘not fun’. If that happens to you, a quick escape to the West Building will lead to fewer people, airy spaces, and the ability to breathe. We didn’t even attempt the Da Vinci exhibit, as that looked even worse (with the additional complication of having to go through a metal detector).

Otherwise, I spent the weekend watching Series 9 of Grange Hill. Yes, the Zammo series. It was probably about a year or so before I really started watching it as a child (I’m guessing ‘87 or ‘88…looking at the wikipedia synopsis, it must have been ‘87), but it’s interesting to see how much gets rattled through alongside the ‘drugs are bad!’ plot that winds through the episodes. We get prejudice against single mothers, underage drinking, ill-advised student-teacher relationships, bullying, foster care issues, open warfare between the staff and governors, threats of strike action over lack of school resources, DANNY KENDALL, dealing with grammar school / secondary modern merging, and yes, the strange person that comes to school on the first day of the fifth year after not shaving all summer. Ahem.

That’s more than you’d see in a lot of ‘adult’ drama serials these days.

Anyhow, a terrible week where we lost David Bowie and Alan Rickman, and I watched Grange Hill and felt old. Plus, all the hot chicken places recommended in Nashville are either shut or close early on Sunday, which is less useful when today is likely the only time I would have been able to go into Nashville itself. Bah.

Suddenly, Nashville

What I didn’t realize about Nashville is that it would be this cold. Seriously, it’s like Boston in March out there (of course, minus the snow). Hot chicken is planned for later in the week, work schedule permitting (ideally, I’d like to go to Hattie B’s this week, and then Prince’s next week, but we shall see).

I’m pausing now to start an amazing new dimension to my travelling life. After over three years of consulting at Mammoth Data, this time, this time, my friends, I have brought tea bags. I will be making a proper cup of tea in my hotel room for the first time.

(yes, the first time it occurred to me in all those years was last night around 11pm. And yes, I did treat it as if it was up there with discovering fire or gravity)

Back now. The lack of a spoon almost led me into stormy waters, but I discovered a plastic straw thing that seemed to work (it did not melt in the tea, so that’s a start). I feel that I need to bring a travel mug to make it a proper cup, though. And that means I can buy a new mug! So this new approach of high living is turning out dividends already.

First restaurant visit of the trip was Local Taco, which sadly made me think that it has been too long since I’ve been to La Vaquita. Not terrible, but not exciting either. Tomorrow, though, a Korean flavour: Soy Bistro. I’m hoping it’s good, because if it is, I’ll likely be eating there several times this week…

Datasets in Spark

The new dataset API lands today in Spark 1.6. Though it’s marked as experimental, it is soon to become an important part of the Spark ecosystem as the platform attempts to push the underlying RDD operations away from the gaze of the everyday developer and into the hands of the Catalyst optimization engine.

Datasets simply present a typed interface over Dataframes, providing a RDD-like way of interacting with the Dataframe with negligible overhead. Creating a dataset is simple:

val boardgameDataframe ="boardgames.json")
case class Boardgame(name: String, rating: Int, type: String)

val  boardgameDataset = boardgameDataframe[Boardgame]

Standard RDD-like operations and aggregations like map, filter, and groupBy are supported. If we want to filter the data so as to only report games rated 8 or above and then group them by type, we’d use the fairly familiar chain below:

val highRanking = boardgameDataset.filter(_.rating >= 8).groupBy(_.type)

A good question at this point is ‘why is this useful, and what advantage does it get me over just doing RDD or dataframe operations?’

Firstly, having a typed interface over dataframes gives you the chance of compile-time type-checking instead of discovering at run-time that the column you’re referencing either doesn’t exist, or is a Long instead of a String (of, course, I’m sure you’re doing testing to make sure this doesn’t happen in production, aren’t you?).

But more importantly, the type information that you provide to a dataset is used to automatically provide an encoder for the Tungsten memory management system (in a similar, but improved manner like Java or Kryo serialization), immediately offering memory benefits over RDDs. As the datasets are based on Dataframes, the execution of your operations is handled by Catalyst, so you’ll benefit from the query optimization engine.

Also, since Catalyst is going to be getting adaptive query optimization (follow SPARK-9850 for updates on this feature, which will allow Spark to dynamically update query plans depending on runtime information), getting used to working with datasets now is going to pay off in your Spark applications in the very near future.

Unitaskers - Accept No Substitutes

Yes, it looks like it has been buried in the dark corner of an Ohio warehouse for forty years.

A photo posted by Ian Pointer (@carsondial) on

But, come on, you can’t argue with the results. And they do get better as the days go on!


A photo posted by Ian Pointer (@carsondial) on

As you can imagine from the photo, they were pretty amazing. I got the recipe from Sprinklebakes, and used a butter / golden syrup / vanilla / yet more sugar mixture for the filling. Beware: looking at 250g of melted butter may induce diabetes. Or at least shame. Try to put those feelings past you and you’ll be rewarded with stroopwafel goodness. And a sugar coma. But it’s a small price to pay.

Since coming back to Durham, I’ve been taking advantage of finally being able to stock up my fridge for the first time since October. This naturally led to cooking all the things, including the aforementioned stroopwafels, a full roast dinner (complete with goose fat potatoes, Yorkshire puddings, and sous-vide beef, just to be completely crazy), apple crumble, pistachio butter, gianduja, and microwave cakes, a white chocolate namelaka…and brown butter ice-cream. Oh, and wontons with spicy chili oil. And a jalfrezi curry with the beef leftovers.

My concern is that I might have burnt myself out.

(only kidding; the Wonder Grinder is a revelation for nut butters - in less than 30 minutes, I had a pistachio butter smoother than anything the Vitamix could ever give me1)

Also, due to a last-minute cancellation, I had my first Escape Room experience this weekend! We escaped the 90s. I complained about the dates for the Spice Girls concert being all wrong and discovered that directional locks are A Thing. There may be more Escape Room antics in the future; apparently there’s one in Columbia too…

Now, though, the long Christmas break is over. I’ve been very lucky that I managed to wrangle a UK-style holiday over the past two weeks, but tomorrow, the year starts anew. Who knows where it’ll take me?2

  1. I am sorry, Vitamix. In this one instance, you failed me. [return]
  2. All accounts indicate Tennessee, but that’s as much as I know at the moment. [return]

Hello 2016

A brand new year deserves a new look. I’ve been meaning to make the site a little more friendly at different resolutions, but I just haven’t had the time. Instead, I took the advantage of the Christmas break to move from Jekyll to Hugo and bashed the Hyde-X theme about somewhat until I was mostly happy.

I’ve moved away from Hoefler Text and back to a sans-serif typeface Open Sans. Other than that, not much has changed; everything is still in Markdown, but oh my does the site rebuild faster in Hugo over Jekyll.

Plans for this year? Yes.

Melted Candles

We’ve reached the tipping point of the holiday where Christmas is over and the flight home is rapidly approaching. I even have to work for a bit tomorrow. Just enough time left for a brief last shop and to melt down the last of the Christmas candles.

A good Christmas, spent like many before, and hopefully many after.

(though, let’s hope fewer computer losses in the upcoming years)

Suddenly, London

It is, unfortunately, easy to get nostalgic for a place that maybe never existed, and if it did, was over as you were being born. But you can feel it as the hills roll past on the way to Haddenham and Thame Parkway, a tiny railway station on the way to Marylebone, traveling past enclosed fields that have likely had their enclosures set for hundreds of years. The seductive call of a Jerusalem that never was. Sheep dogs being trained on the hills

You come back once a year and you can feel the country changing. The creeping Americanisms, your home town moving on and building itself anew for the 21st century, ripping old parts out with no thought as to the memories you made in them. After all, you’ve gone now. It’s not your place anymore.

And yet, the muscle memory of knowing exactly when to turn, of the cottages racing by, replaced by the terraces; the post-war consensus survivors smiling at you and whispering ‘it wasn’t a dream, we did this, once’. The modern shoebox estates, where everything is crammed together and the green disappears. The towers as London approaches, much fewer in number, but Trellick still standing proud, brutal, and at long last, loved.

Walking up the steps to the Royal Festival Hall, Saint Etienne’s ‘London Belongs To Me’ coming on random play, like the iPhone just knows, big smile as the Queen Elizabeth Hall and Hayward Gallery come into view and the book sellers are putting out their tables. And walking up straight into a group of Pelican books. They knew, too.

But even as the Tate Modern’s huge chimney breaks out in front of you, there’s the realization - this place meant and means so much to you, and you’re happy to be back, but it’s not yours anymore. You don’t belong here in in the same way you did even five years ago. You can visit, but London will never belong to you. The construction cranes lining the sky across the water announce that it will likely not belong to your friends here for that long, either.

Saying all that, had a good time in London yesterday.

Suddenly, Britain


It takes a bit of getting used to being back in a bedroom that is literally smaller than my bathroom, and I’m not entirely convinced that the sun even came up today, but it’s good to be back home.

I’m not entirely sure that my sister feels exactly the same way, seeing as how I’ve spent the past 48 hours wearing a Father Christmas hat and making sure that she’s full of festive cheer. I have to hide the knives1.

A week of work awaits, then a quick trip to London (my first in two years!!), and finally: CHRISTMAS!

plays Slade very loudly

  1. though she probably already scored the biggest burn of the holiday by saying that listening to my music is like being trapped in Phonogram. [return]