Finally, Durham

It has been a long time. But I am currently about to spend my second night in Durham for what seems a long, long time.

Meanwhile, we may trace the import of American Prom Culture to the UK to Grange Hill in 1989. Or thereabouts.

(yes, past the Death of Danny Kendall now)

Second week of San Francisco (okay, apparently, I’m legally required to point out that it was Newark, just in case people get the wrong idea) was much quieter than the first, mainly because I was working onsite, but partly because I fell ill around Tuesday evening.

It is nice to be home (in the Durham one, anyway), but I spent most of Saturday in a tired haze, and it’s the end of Sunday right now. A weekend pretty much lost.

But, look on the bright side, I guess: a week in Durham! Complete with a Spark Meetup on Tuesday night, and then a weekend of finally getting back into the Chocolate Room and making the first major batch of chocolates for the year.

Talk Of The Town

It has been an odd week. My one-week trip to San Francisco suddenly became two, with a week-long preamble before the actual engagement. So I have spent the week in a hotel lobby working during the day, and then sometimes braving BART in the evening for a quick trip into the city.

Of course, I’m not quite in San Francisco itself, but Newark, which I didn’t know existed before Sunday evening (natives of the Bay Area point out that is a good thing, and really, I should have remained in that blissful state of ignorance). Exploration of the city itself was therefore put mostly on hold until the weekend.

But what a weekend! Saturday was split into two - firstly, the traditional ‘Ian walks for miles and miles and is then surprised that his foot hurts’ approach. This involved a quick breakfast of macarons, a voyage to Dandelion Chocolate (they have a brownie flight, which seems the most SF-thing ever), a disastrous attempt to visit Costco (on a Saturday afternoon no less - what was I thinking?), a pilgrimage to DNA Lounge, and then the long, long walk to the where I was staying. Turns out that they weren’t joking about the hills. Not one bit.

Having had something of a rest, it was time for the second part of the day. I have a new friend! Winebratsf of LushiousLushes had spent part of the week taunting me with the abundance of dark chocolate digestives in San Francisco, but also suggested that she could show me some edited highlights of the city. Which mainly included gin (Whitechapel), whiskey (Whiskey Thieves), with a brief stop for food and stocking up on spirits that aren’t on the NC ABC list.

Eventually, after a merry night (and learning lots about American whiskey!), I staggered back home . Or rather, continued my week-long foray into the world of ride-sharing. I can see why it’s popular. They gave me $50 of free credit for signing up, and it was pretty easy to get places using it. That didn’t stop me from feeling guilty about it, mind you. I wonder if the geospatial / communications tech could be commoditized and passed around to local taxi companies.

Sunday! Brunch! More walking! And then back to the hotel. With ibuprofen to help my poor foot. Sadly, I had to skip Oakland this time, but hopefully, I’ll be back!

San Francisco


A photo posted by Ian Pointer (@carsondial) on

Brownie flight.

A photo posted by Ian Pointer (@carsondial) on


A photo posted by Ian Pointer (@carsondial) on

Did I really just walk all that way? (Yes)

A photo posted by Ian Pointer (@carsondial) on


A photo posted by Ian Pointer (@carsondial) on

The legally-required photo of a tram in SF.

A photo posted by Ian Pointer (@carsondial) on

Om nom nom

A photo posted by Ian Pointer (@carsondial) on

Suddenly, Columbia???

I’m writing this from the second gate change in Charlotte airport waiting for a flight to San Francisco, haven been driven up by Tammy from Columbia. Now, my intended weekend was to relax after a rather stressful deployment to Nashville1 and then roll into work on Monday to finish off a project that has been hanging over me for…oooh, about three months.

The snowstorm put a pretty stop to that. Instead of flying home on a leisurely basis on Friday afternoon, I had to rush back to the airport in Nashville on Thursday night…but all the flights to RDU had already been cancelled, so instead I found myself flying to Columbia, SC, to spend a rather impromptu visit (and I think I caused sadness by not having brought any board games with me. So, a note: along with the passport, always carry an emergency board game). It was a fun weekend! Just not the one I was expecting.

And instead of gingerly making my way back to Durham today, it’s San Francisco. My first-ever visit to Silicon Valley. This has the potential to be exciting, but I can’t help but feel that it would have been better if it was next week instead of right now…

  1. ask me sometime about how I managed to avoid re-writing the Spark Thrift Server by writing a one-line Scala function that turns a Spark Dataframe into a Hive DDL string and some hasty Hive Metastore schema changes. It’s exciting! Or…not. [return]

An Infinite Tiling of People

After a bad week where it seemed like a Hadoop cluster generated every possible problem (up to and including spontaneously generating new cluster IDs for the namenode and datanodes), it was nice to be back in Durham, even only for a short time.

But, no time for love, Dr. Jones, as after waking up on Saturday morning, it was time to re-pressurize my back right tyre and head all the way to Raleigh where I met up with Tammy, Robert, the two smaller humans, and Christie for a visit to the Escher exhibit at the NC Museum of Art. All this time I’ve lived here, and this was the first time I’ve visited!

I can definitely recommend catching the exhibit in the upcoming (and last week), with one tiny caveat: if you don’t like crowds, try and go during the weekday if possible. There were so many people around that it did at times tip over into being ‘not fun’. If that happens to you, a quick escape to the West Building will lead to fewer people, airy spaces, and the ability to breathe. We didn’t even attempt the Da Vinci exhibit, as that looked even worse (with the additional complication of having to go through a metal detector).

Otherwise, I spent the weekend watching Series 9 of Grange Hill. Yes, the Zammo series. It was probably about a year or so before I really started watching it as a child (I’m guessing ‘87 or ‘88…looking at the wikipedia synopsis, it must have been ‘87), but it’s interesting to see how much gets rattled through alongside the ‘drugs are bad!’ plot that winds through the episodes. We get prejudice against single mothers, underage drinking, ill-advised student-teacher relationships, bullying, foster care issues, open warfare between the staff and governors, threats of strike action over lack of school resources, DANNY KENDALL, dealing with grammar school / secondary modern merging, and yes, the strange person that comes to school on the first day of the fifth year after not shaving all summer. Ahem.

That’s more than you’d see in a lot of ‘adult’ drama serials these days.

Anyhow, a terrible week where we lost David Bowie and Alan Rickman, and I watched Grange Hill and felt old. Plus, all the hot chicken places recommended in Nashville are either shut or close early on Sunday, which is less useful when today is likely the only time I would have been able to go into Nashville itself. Bah.

Suddenly, Nashville

What I didn’t realize about Nashville is that it would be this cold. Seriously, it’s like Boston in March out there (of course, minus the snow). Hot chicken is planned for later in the week, work schedule permitting (ideally, I’d like to go to Hattie B’s this week, and then Prince’s next week, but we shall see).

I’m pausing now to start an amazing new dimension to my travelling life. After over three years of consulting at Mammoth Data, this time, this time, my friends, I have brought tea bags. I will be making a proper cup of tea in my hotel room for the first time.

(yes, the first time it occurred to me in all those years was last night around 11pm. And yes, I did treat it as if it was up there with discovering fire or gravity)

Back now. The lack of a spoon almost led me into stormy waters, but I discovered a plastic straw thing that seemed to work (it did not melt in the tea, so that’s a start). I feel that I need to bring a travel mug to make it a proper cup, though. And that means I can buy a new mug! So this new approach of high living is turning out dividends already.

First restaurant visit of the trip was Local Taco, which sadly made me think that it has been too long since I’ve been to La Vaquita. Not terrible, but not exciting either. Tomorrow, though, a Korean flavour: Soy Bistro. I’m hoping it’s good, because if it is, I’ll likely be eating there several times this week…

Datasets in Spark

The new dataset API lands today in Spark 1.6. Though it’s marked as experimental, it is soon to become an important part of the Spark ecosystem as the platform attempts to push the underlying RDD operations away from the gaze of the everyday developer and into the hands of the Catalyst optimization engine.

Datasets simply present a typed interface over Dataframes, providing a RDD-like way of interacting with the Dataframe with negligible overhead. Creating a dataset is simple:

val boardgameDataframe ="boardgames.json")
case class Boardgame(name: String, rating: Int, type: String)

val  boardgameDataset = boardgameDataframe[Boardgame]

Standard RDD-like operations and aggregations like map, filter, and groupBy are supported. If we want to filter the data so as to only report games rated 8 or above and then group them by type, we’d use the fairly familiar chain below:

val highRanking = boardgameDataset.filter(_.rating >= 8).groupBy(_.type)

A good question at this point is ‘why is this useful, and what advantage does it get me over just doing RDD or dataframe operations?’

Firstly, having a typed interface over dataframes gives you the chance of compile-time type-checking instead of discovering at run-time that the column you’re referencing either doesn’t exist, or is a Long instead of a String (of, course, I’m sure you’re doing testing to make sure this doesn’t happen in production, aren’t you?).

But more importantly, the type information that you provide to a dataset is used to automatically provide an encoder for the Tungsten memory management system (in a similar, but improved manner like Java or Kryo serialization), immediately offering memory benefits over RDDs. As the datasets are based on Dataframes, the execution of your operations is handled by Catalyst, so you’ll benefit from the query optimization engine.

Also, since Catalyst is going to be getting adaptive query optimization (follow SPARK-9850 for updates on this feature, which will allow Spark to dynamically update query plans depending on runtime information), getting used to working with datasets now is going to pay off in your Spark applications in the very near future.

Unitaskers - Accept No Substitutes

Yes, it looks like it has been buried in the dark corner of an Ohio warehouse for forty years.

A photo posted by Ian Pointer (@carsondial) on

But, come on, you can’t argue with the results. And they do get better as the days go on!


A photo posted by Ian Pointer (@carsondial) on

As you can imagine from the photo, they were pretty amazing. I got the recipe from Sprinklebakes, and used a butter / golden syrup / vanilla / yet more sugar mixture for the filling. Beware: looking at 250g of melted butter may induce diabetes. Or at least shame. Try to put those feelings past you and you’ll be rewarded with stroopwafel goodness. And a sugar coma. But it’s a small price to pay.

Since coming back to Durham, I’ve been taking advantage of finally being able to stock up my fridge for the first time since October. This naturally led to cooking all the things, including the aforementioned stroopwafels, a full roast dinner (complete with goose fat potatoes, Yorkshire puddings, and sous-vide beef, just to be completely crazy), apple crumble, pistachio butter, gianduja, and microwave cakes, a white chocolate namelaka…and brown butter ice-cream. Oh, and wontons with spicy chili oil. And a jalfrezi curry with the beef leftovers.

My concern is that I might have burnt myself out.

(only kidding; the Wonder Grinder is a revelation for nut butters - in less than 30 minutes, I had a pistachio butter smoother than anything the Vitamix could ever give me1)

Also, due to a last-minute cancellation, I had my first Escape Room experience this weekend! We escaped the 90s. I complained about the dates for the Spice Girls concert being all wrong and discovered that directional locks are A Thing. There may be more Escape Room antics in the future; apparently there’s one in Columbia too…

Now, though, the long Christmas break is over. I’ve been very lucky that I managed to wrangle a UK-style holiday over the past two weeks, but tomorrow, the year starts anew. Who knows where it’ll take me?2

  1. I am sorry, Vitamix. In this one instance, you failed me. [return]
  2. All accounts indicate Tennessee, but that’s as much as I know at the moment. [return]

Hello 2016

A brand new year deserves a new look. I’ve been meaning to make the site a little more friendly at different resolutions, but I just haven’t had the time. Instead, I took the advantage of the Christmas break to move from Jekyll to Hugo and bashed the Hyde-X theme about somewhat until I was mostly happy.

I’ve moved away from Hoefler Text and back to a sans-serif typeface Open Sans. Other than that, not much has changed; everything is still in Markdown, but oh my does the site rebuild faster in Hugo over Jekyll.

Plans for this year? Yes.

Melted Candles

We’ve reached the tipping point of the holiday where Christmas is over and the flight home is rapidly approaching. I even have to work for a bit tomorrow. Just enough time left for a brief last shop and to melt down the last of the Christmas candles.

A good Christmas, spent like many before, and hopefully many after.

(though, let’s hope fewer computer losses in the upcoming years)