Flame Graphs With Spark

I couldn’t sleep last night, so after I gave up lying in bed at 6:58am, I decided to write something that I’ve been meaning to create for a while.

Hence: spark-flame. A way of quickly obtaining flame graphs on Apache Spark worker nodes. It’s a short Ansible playbook that takes a YARN application id, does some terrible things with ps to get the java process ids for the executors, run perf, get the JVM symbols using perf-map-agent, generate the flame graphs and copy them back to your local machine.

The steps the playbook takes are detailed in Brendan Gregg’s Java In Flames blog. All it really does is follow those steps whilst making sure that the workers have a copy of the perf-map-agent libraries and finally copying the resulting flame graphs back down to your local box.

There’s a couple of options to tweak; you can alter the length of the perf sample recording, and you can change the options to perf itself (currently it’s set to -ag -F 997, so it’ll sample at 997Hz across all CPUs and generate call graph stacks). But that’s about it!

And as a bonus, here’s a sample SVG from a worker doing a map over 10m doubles:

Tired, Part 2

More news coming next week.

I’m just so tired at the moment. Tired of lots of things. Thankfully, not everything.

I apologize for a lacklustre August. Hopefully September will be better. Hopefully I will be better.

Tired

Well.

Five Years Later

It doesn’t quite seem like five years.

I’m not in the happiest frame of mind right now due to an ongoing mouse issue that I can only deal with in the 40 or hours that I’m home before travelling to a remote office park again for the week.

But, aside from that right now, it’s been a good five years. Old friends, lots of wonderful new friends, achieving some childhood dreams (working for Activision!), navigating the weird and wonderful world of the United States, and more besides.

I understand that as an immigrant to this country, I’m playing things on easy mode. Whilst many foreign accents are greeted with defensiveness and hostility, being a White British Male means that instead I’m greeted with ‘Oh my goodness, I love that accent!’, even at times propositioned in a style straight out of Love Actually. So my experience isn’t the same as many. But it has been mostly good so far.

Living here has also made me so much more British. I mean, I’m writing this while wearing a British Rail t-shirt. Being disconnected from home has made me appreciate the different environment I grew up in (both from a national and a personal front). Look, I watched a Morecambe and Wise Christmas Special last week. I have a problem.

Anyway. Five years. Ups and downs. But hopefully more ups to come.

Blackcurrants RIP

Just before I left for St. Louis last week, I attempted to trim the hedges down at the far edge of the vast expanse that is my back garden (okay, it’s not that big, but there’s a lot of grass, trees, bushes, and apparently snakes. I would prefer a large quantity of concrete).

Now, I thought it didn’t go too badly, considering it was the first time I ever used a hedge trimmer. I kept all my fingers and the hedges were cut back enough that they wouldn’t interfere with people waiting by the bus stop.

However, my neighbour across the road had other ideas. He’d been keeping an eye on the people that had been cutting my grass for the last couple of weeks and said that he didn’t think they were doing a good job. He was willing to cut the grass and also cut back the hedges a bit more.

Fine, I thought. It might be nice to have some one who knows what they’re doing to tidy up my hedge trimming. I went off to St. Louis Sunday, coming back late on Thursday night. While I was making the celebratory ‘returning home’ cup of tea1, I looked out the window.

The hedges were gone. Not just trimmed, but cut down and ripped out.

Hurrah! One fewer thing to worry about in the back garden. He’s coming around for the tree next to the kitchen next week.

Unfortunately, there was a casualty in the hedge-felling. My illicit blackcurrants were mistaken for weeds. So they are no longer with me. Sadness.2

Today? Well, today I’m heading to St. Louis again. This time, I hope to see a little of the city itself…


  1. I am nothing but a stereotype. [return]
  2. There is a long-term plan for blackcurrants, but I’m not at liberty to speak of it just yet. [return]

All Aboard To St. Louis!

Just to make it clear, as apparently I confused and worried a few people: no, I am not moving to St. Louis! But I will be starting a job tomorrow with Xavient Information Systems which will involve me travelling to St. Louis occasionally and working from home at other times.

And now I have to run to find some power, so consider this a very short update. But, hurrah - new job!

Holding Patterns

I watched the entire RNC, sober. I think that qualifies for some sort of reward. And the reward is to watch the entire DNC! Hurrah! Political masochist right here, you see. At least next week is likely to have less fire and brimstone.

Oh, also my first dental work in the USA! Go me! And more tomorrow. You’d think I’m trying to get it all done in a short time-frame, wouldn’t you? Well—[REDACTED. Nice try — Ed.]

Meanwhile, I have a mice problem. I am acutely embarrassed by this, as I tend to veer close to the OCD side of cleanliness in the kitchen. Yet they are here, and they seem to be here in numbers. There have been ultrasonic devices, bait traps, and as of Friday, exterminators, yet they keep coming back.

It’s starting to wear me down, to be honest. I have almost zero desire to cook in that kitchen, I’m beginning to actively hate the house, and at the moment I have little to take my mind of it except the downfall of the Republic and the West.

Which might explain why I’ve spent today building time-adaptive versions of count-min sketch for Rust, Ruby, and Elixir. They’re all a touch rough at the moment, but I should have them up on GitHub in the next couple of weeks. It’s nice to be able to read a paper, think ‘oh, I could knock that up’, and actually do it the next day.

I’m hoping that the coming week will make things less uncertain. Let us hope, anyhow.

Not A Kinder Egg

An unexpected last-minute trip to Cincinnati, then. And things happened!

Firstly, a return to Jungle Jim’s. Except, this was a totally different location. There’s more than one! (okay, there’s only two. But still, that’s a lot of supermarket). This one was a touch smaller, but still had its fair share of surprises. One such surprise was the almost-complete absence of Cadbury products as opposed to the other location. I wonder if Hershey got to them?

And here was the other surprise, in the Germany department:

A photo posted by Ian Pointer (@carsondial) on

(apologies; it got a little squished in the suitcase)

Does it remind you of anything? Anything at all?

And here’s the inside:

Not a Kinder egg!

A photo posted by Ian Pointer (@carsondial) on

Which contains, in disassembled form:

A photo posted by Ian Pointer (@carsondial) on

There you have it. If you have a deep need for Kinder Eggs, Cincinnati can get you hooked up for a knock-off at least. Just don’t expect wonders from the ‘chocolate’ shell.

(seriously, I can still taste the horror an hour later)

Anyhow, when I last went to the area, we discovered that a new board game café, The Rook, was opening the week after I went home. Curses! But! Return trip!

It’s not bad! I haven’t been to Thirsty Meeples in Oxford (my excuse is that it opened after I left the UK), but I imagine it’s a similar set up - drinks, food, and board games. Oh, so many board games. Everything from Guess Who to Terra Mystica.

A word of advice, though: don’t attempt to learn Terra Mystica in the middle of a loud board game café. It will lead to sadness and misery. May I suggest Cyclades instead? Fairly easy enough to pick up, a good game, and one of the proprietors of the shop may compliment the pair of you on your choice!

(which is good, because I felt bad for suggesting Terra Mystica to Tammy in the first place!)

All this and the heart warming tale of an Septapus and his three hearts. What more could a weekend need (it could do with less of pins and needles all over my body and more of getting sleep, but apparently that wasn’t on the agenda). And! Being able to go outside for more than five minutes and not hating my choices in life! Heat without insane humidity! It’s crazy!

Tomorrow: the dentist. Run away!

The Three-Day-Wasp-War

They came on Thursday morning. Three of them, occupying the hallway, the chocolate room and the kitchen. My standard procedure in the case of a wasp infiltration is to hide until the wasp goes away or gets so sleepy that I can drop a container on it (see also: England Made Me).

This time, though, it was a problem. Three of them at once, none of them sleepy in the slightest, and all seeming quite angry with the world at large.

I did not shrink from my destiny. I knew what I had to do to restore order to this house.

I promptly barricaded myself into my bedroom, stopped all the gaps underneath the doors and prepared to wait them out.

I am 37 years old.

Friday morning came and went…but they weren’t leaving, and they weren’t dying. I escaped from the bedroom and went off to Home Depot to buy supplies. But while I found all sorts of sprays to destroy a wasps’ nest, I overlooked getting something that would work inside to kill them.

So I spent Friday night in my bedroom, slipping out at times to get drinks. To be fair, they were getting a little sleepier at this time, but still not exactly willing to be approached with containers for extraction.

By Saturday morning, it was getting silly. After being pointed to exactly the right to of spray to obtain, I made my third visit to Home Depot in three days, loaded myself up with Raid and came back to reclaim the house.

Two of the wasps fell easily, but the third had vanished. I looked around the kitchen in vain and then went on with my day.

Five hours later, I was emptying the dish washer and I caught it in the corner of my eye, wandering around the sink (as is traditional, I jumped about five miles in the air, and had I been holding a chicken leg, I would have caused some serious damage1). Now, I couldn’t spray into the sink, and the wasp was happily sitting on something that made it impossible to drop a container down and prevent it from flying off.

There was only one thing to do. One terrible thing. I gently tilted the kitchen tap in its direction, switched it to full spray, and then turned the tap on full.

I drowned a wasp and sent it down the sink.

I’m not proud. But I have my kitchen back, after a fashion.

All this and Portugal won Euro 2016. A terrible week, all told.


  1. There’s about ten of you that will get this reference. You’re welcome. [return]

Self-Promotional Post

As of today, I’m looking for a new position. If you’re looking for somebody to do Big Data / Spark / Storm / Scala / Hadoop / Ruby / Elixir / Docker / Mesos / AWS work, then I’m available! Ideally, I’m looking for a remote position, or otherwise in Durham (downtown very much preferred).

Some people have been asking, so just to point out that I’m also available for consultancy on any of the above.

Here’s my extended resumé, and you can reach me at ian@snappishproductions.com.