Just One More Thing

After careful consideration, and around thirty episodes later, I think I don’t like the original 70s era of Columbo all that much. My mind keeps comparing it to The Sweeney, broadcast on ITV roughly around the same time…and, okay, maybe it’s not a fair comparison, but Columbo is slow, plodding, and not much fun. Even a cruise ship episode guest-starring Robert Vaughn, Patrick Macnee, and Dean Stockwell (with amazing bushy eyebrows) involving a comedy routine where Mrs. Columbo was just out of view every five minutes struggled to rise above ‘interesting’. Falk is good, obviously, but removing the structure of ‘whodunnit’ means that instead of suspense and a guessing game, you get to watch Columbo irritate the murderer for an hour.

I’m guessing it played better in weekly episodes.

Look, I’m still not entirely well, and the nights are long.

The Last Big Weekend of The Summer

Last week, I left you with a couple of posts on flame graphs and that was your lot (BUT HEY, somebody might find them useful!). However, in the background, many things happened. I’ve started a new position at Kogentix, and I spent last Friday getting divorced (a relatively painless procedure involving a somewhat sassy judge).

I had fully prepared to spend Labor Day Weekend locked up in my room watching Thames TV idents from the 1970s (I do have worrying form for this, you know), but thankfully that was not to be. Tammy made a surprise visit to the area and that led to food trucks, baking, and time-travelling bigamy. What more could you ask for in a Bank Holiday1 weekend?

If I move away from this area, I will definitely miss the food truck rodeos. We have them so often that it’s easy to get jaded; indeed I don’t think I’ve been to one for over a year. I remember when it was just 13 trucks and everybody was excited about the ill-fated Grilled Cheese Bus. These days, it’s over 50, including Food Network winners and people attempting to charge $14 for three tacos. Ah, Durham.

Then more people! We met up with Christie and Ashley and had a great day wandering the rodeo, eating everything from fried cauliflower to bulkogi cooked in Cheerwine. And then my first visit to Ponysaurus, which seems like a great place and if I actually drank beer, I’d probably go more, considering it’s about four minutes away from home (GENTRIFICATION IS COMING).

Although I did not spend the weekend watching idents, there was a theme of British television. Just somewhat more recent. People watched Very British Problems and laughed as they recognized me far too often, along with me having to point out who Vic Reeves is. Which led to me doing a grand showing of this classic segment of television:

After that, it was time to catch up on the new series of Great British Bake-Off, whereupon I shouted loudly at the TV. “HAVE YOU NEVER HONESTLY NEVER SEEN A JAFFA CAKE IN YOUR LIFE?” may have been a refrain during certain parts.

And then. “Would you mind if we watched this British comedy?” Given that the last time I suggested something for a group to watch, it ended up being Jubilee, I am astounded that people said yes. And that’s how I ended up having to explain the backstory of Goodnight Sweetheart to an incredulous American audience. “Wait, the main character is a bigamist? And this was a popular show in the 1990s?”

Britain: A Strange Country.

Anyway. Yes, it was corny. Yes, about 73% of the jokes fell flat while the studio audience sounded like they had been exposed to nitrous oxide for a few hours before taping. Yes, the show deeply misses Dervla Kirwan2. And yet, the revival had something. The mixture of the past catching up in the 60s with the plagiarized songs being released and the cultural shock of 2016 was interesting. Perhaps I relate to the cultural shock of returning home every year to find things have changed in subtle and not-so-subtle ways each time. In the case of Bicester, that normally means more houses, but sometimes you go back to find out that the bus stop you used for around seven years to get to work has turned into a giant Sainsbury’s and when I get home this Christmas the Big Tesco will have gone…replaced by the visage of a Tesco Extra standing on the countryside like a monolith from 2001. Or at least that’s what I’m told, anyhow. So, despite all the corniness, I’d be interested in seeing another series of that.

So, a great last Summer weekend. Unfortunately, I’ve spent this week drinking Lemsip by the case and fighting some sort of infection. It is mostly cleared now, but I’ve gone through three boxes of tissues in less than two days. Hopefully next week will finally see the back of it!

  1. I spelt it ‘Labor’ Day. Let me have this. [return]

Flame Graphs in Rust on Linux

I’ve been working on a few toy Rust programs and libraries of late. One of these is dolby, an implementation of Adaptive Count-Min Sketch as detailed in this paper from Microsoft Research. CMS data structures are designed to deal with the problem of counting things at scale - when your incoming data stream is (essentially) infinite, how can you reason about the data using only finite resources? CMS algorithms and structures get around that by using probability - they can give you approximate answers.

Running an algorithm like this at scale is likely to amplify any inefficiencies in the implementation, and knocking off a few seconds, milliseconds, or even microseconds off the running time of a loop or other section of the program can yield massive rewards. And a great way to identify potential improvements or discovering problematic issues is with flame graphs.

(The following was all done on a Ubuntu 16.04 machine, but don’t worry! I’m not using any fancy BPF tricks or anything - as long as you can run a recent-ish perf on your machine (mine is 4.4.15), you should be able to replicate this fairly easily.)

In order to get useful information out of our profiling tools, we need to tell the Rust compiler (rustc) to include DWARF debugging symbols in the binary it is going to create. To do this, we add a small section to our cargo.toml file:

    debug = true

Once we have built the binary with cargo build, we can do a system-wide profile using perf. Here I’m running a test against 10 million entries being added to a dolby data structure:

    root# perf record -ag  ~/dolby/target/debug/dolby

(the -a flag ensures profiling capture on all CPUs, the -g flag switches on stack-chain recording, which is needed to build the flame graph structure)

After that has finished, we can use Brendan Gregg’s flame graph tools to convert the output into graphical form.

    perf script | ~/FlameGraph/stackcollapse-perf.pl | ~/FlameGraph/flamegraph.pl > rust.svg

Opening rust.svg in a browser gives us a fancy flame graph!


Looking at the graph, it seems that the majority of the time dolby is on-CPU, it is running the insert method. Which makes sense, seeing as the test harness is firing 10m numbers into the structure! We can also see by walking up the chain that insert spends most of its time in the hash_index method, and most of the samples inside hash_index are actually while Rust is doing hashing calculations within the murmurhash3 crate. Changing the hash function to something less expensive than murmurhash3 may therefore result in improved insert performance.

Anyhow - flame graphs in Rust! They’re easy! Use them!

Flame Graphs With Spark

I couldn’t sleep last night, so after I gave up lying in bed at 6:58am, I decided to write something that I’ve been meaning to create for a while.

Hence: spark-flame. A way of quickly obtaining flame graphs on Apache Spark worker nodes. It’s a short Ansible playbook that takes a YARN application id, does some terrible things with ps to get the java process ids for the executors, run perf, get the JVM symbols using perf-map-agent, generate the flame graphs and copy them back to your local machine.

The steps the playbook takes are detailed in Brendan Gregg’s Java In Flames blog. All it really does is follow those steps whilst making sure that the workers have a copy of the perf-map-agent libraries and finally copying the resulting flame graphs back down to your local box.

There’s a couple of options to tweak; you can alter the length of the perf sample recording, and you can change the options to perf itself (currently it’s set to -ag -F 997, so it’ll sample at 997Hz across all CPUs and generate call graph stacks). But that’s about it!

And as a bonus, here’s a sample SVG from a worker doing a map over 10m doubles:

Tired, Part 2

More news coming next week.

I’m just so tired at the moment. Tired of lots of things. Thankfully, not everything.

I apologize for a lacklustre August. Hopefully September will be better. Hopefully I will be better.



Five Years Later

It doesn’t quite seem like five years.

I’m not in the happiest frame of mind right now due to an ongoing mouse issue that I can only deal with in the 40 or hours that I’m home before travelling to a remote office park again for the week.

But, aside from that right now, it’s been a good five years. Old friends, lots of wonderful new friends, achieving some childhood dreams (working for Activision!), navigating the weird and wonderful world of the United States, and more besides.

I understand that as an immigrant to this country, I’m playing things on easy mode. Whilst many foreign accents are greeted with defensiveness and hostility, being a White British Male means that instead I’m greeted with ‘Oh my goodness, I love that accent!’, even at times propositioned in a style straight out of Love Actually. So my experience isn’t the same as many. But it has been mostly good so far.

Living here has also made me so much more British. I mean, I’m writing this while wearing a British Rail t-shirt. Being disconnected from home has made me appreciate the different environment I grew up in (both from a national and a personal front). Look, I watched a Morecambe and Wise Christmas Special last week. I have a problem.

Anyway. Five years. Ups and downs. But hopefully more ups to come.

Blackcurrants RIP

Just before I left for St. Louis last week, I attempted to trim the hedges down at the far edge of the vast expanse that is my back garden (okay, it’s not that big, but there’s a lot of grass, trees, bushes, and apparently snakes. I would prefer a large quantity of concrete).

Now, I thought it didn’t go too badly, considering it was the first time I ever used a hedge trimmer. I kept all my fingers and the hedges were cut back enough that they wouldn’t interfere with people waiting by the bus stop.

However, my neighbour across the road had other ideas. He’d been keeping an eye on the people that had been cutting my grass for the last couple of weeks and said that he didn’t think they were doing a good job. He was willing to cut the grass and also cut back the hedges a bit more.

Fine, I thought. It might be nice to have some one who knows what they’re doing to tidy up my hedge trimming. I went off to St. Louis Sunday, coming back late on Thursday night. While I was making the celebratory ‘returning home’ cup of tea1, I looked out the window.

The hedges were gone. Not just trimmed, but cut down and ripped out.

Hurrah! One fewer thing to worry about in the back garden. He’s coming around for the tree next to the kitchen next week.

Unfortunately, there was a casualty in the hedge-felling. My illicit blackcurrants were mistaken for weeds. So they are no longer with me. Sadness.2

Today? Well, today I’m heading to St. Louis again. This time, I hope to see a little of the city itself…

  1. I am nothing but a stereotype. [return]
  2. There is a long-term plan for blackcurrants, but I’m not at liberty to speak of it just yet. [return]

All Aboard To St. Louis!

Just to make it clear, as apparently I confused and worried a few people: no, I am not moving to St. Louis! But I will be starting a job tomorrow with Xavient Information Systems which will involve me travelling to St. Louis occasionally and working from home at other times.

And now I have to run to find some power, so consider this a very short update. But, hurrah - new job!

Holding Patterns

I watched the entire RNC, sober. I think that qualifies for some sort of reward. And the reward is to watch the entire DNC! Hurrah! Political masochist right here, you see. At least next week is likely to have less fire and brimstone.

Oh, also my first dental work in the USA! Go me! And more tomorrow. You’d think I’m trying to get it all done in a short time-frame, wouldn’t you? Well—[REDACTED. Nice try — Ed.]

Meanwhile, I have a mice problem. I am acutely embarrassed by this, as I tend to veer close to the OCD side of cleanliness in the kitchen. Yet they are here, and they seem to be here in numbers. There have been ultrasonic devices, bait traps, and as of Friday, exterminators, yet they keep coming back.

It’s starting to wear me down, to be honest. I have almost zero desire to cook in that kitchen, I’m beginning to actively hate the house, and at the moment I have little to take my mind of it except the downfall of the Republic and the West.

Which might explain why I’ve spent today building time-adaptive versions of count-min sketch for Rust, Ruby, and Elixir. They’re all a touch rough at the moment, but I should have them up on GitHub in the next couple of weeks. It’s nice to be able to read a paper, think ‘oh, I could knock that up’, and actually do it the next day.

I’m hoping that the coming week will make things less uncertain. Let us hope, anyhow.