Autoforwarding Security Credentials In Storm

Using Kerberos with Storm is, like most things involving Kerberos, an experience akin to pulling teeth with a pair of tweezers: it hurts and it goes on for a long time. Can you get the keytabs generated and into the right place, and what does that end up meaning for your Storm supervisor nodes? Wouldn’t it be lovely if Storm could simply hand out Hadoop Kerberos credentials to a topology when it is submitted and Everything Just Works™?

Well, if you’re attempting to use HBase or HDFS in your Bolts, then things are looking up for you. You can use the AutoHBase and AutoHDFS classes to do exactly that, and then the only keytab you need worry about is the one on your Nimbus server.

Except

It’s never quite that easy. Mainly, the thing you have to be aware of is this: the class hierarchy of AutoHDFS and AutoHBase have changed in the last few months, so if you’re using a platform like Cloudera, MapR, or HortonWorks, you may find yourself staring at a terminal wondering why on Earth Kerberos isn’t working…and like all things Kerberos, the errors are obtuse and unhelpful.

Anyway, the old hierarchy is:

backtype.storm.security.auth.hadoop.AutoHDFS
backtype.storm.security.auth.hadoop.AutoHBase

and the new locations are:

org.apache.storm.hdfs.common.security.AutoHDFS
org.apache.storm.hdfs.common.security.AutoHBase

Then, in your topology, update the Config.TOPOLOGY_AUTO_CREDENTIALS with a list of all the credentials it needs access to (in this example, just HDFS, but you could simply add HBase into the autoCreds list and it’ll have access to HBase too:

public static void main(String[] args) throws Exception {    
    //...
    Config cfg = new Config();
    List<String> autoCreds= new ArrayList<String>();
    
    // Use this hierarchy for an older distribution, e.g. HDP 2.2
    autoCreds.add("backtype.storm.security.auth.hadoop.AutoHDFS");
    // This is the current hierarchy
    //autoCreds.add("org.apache.storm.hdfs.common.security.AutoHDFS");
    cfg.put(Config.TOPOLOGY_AUTO_CREDENTIALS, autoCreds);
    
    // [...other topology and config setup...]
    
    StormSubmitter.submitTopology(TOPOLOGY_NAME, cfg, builder.createTopology());
  }

Then, on your Nimbus server, you need to update your storm.yaml (this example uses the current hierarchy, but you can replace the entries with the old ones and it’ll work if you’re on a non-current version of Storm):

nimbus.autocredential.plugins.classes: [“org.apache.storm.hdfs.common.security.AutoHDFS”, “org.apache.storm.hdfs.common.security.AutoHBase”] 

nimbus.credential.renewers.classes: [“org.apache.storm.hdfs.common.security.AutoHDFS”, "org.apache.storm.hdfs.common.security.AutoHBase”]

hdfs.keytab.file: "/path/to/keytab/on/nimbus" 
hdfs.kerberos.principal: "superuser@EXAMPLE.com" 
nimbus.credential.renewers.freq.secs : 82800

Restart your Nimbus server, submit your topology and watch Secure HDFS be authenticated without any further Kerberos nightmares! This time, at least. Kerberos is always out there, waiting. Waiting.

Avengers 2 - Getting Too Old For This

Avengers: 2, then:

Things I liked:

  • An actual rescue and evacuation of a city area where superpowers came in handy for situations other than hitting things!
  • Grafting Ultron onto Stark makes a good shortcut (having to explain that, no, really, Ant-Man created in the MU made me look slightly silly)

And things that were somewhat less liked:

  • Stark ruins everybody’s day and then saves the day by doing exactly the same thing (I imagine this will crop up again during Civil War, but seems somewhat lazy).
  • After making such a big deal about why Pietro and Wanda really hate Stark, there’s no scene with them just having a general chat about how he supplied arms that killed their parents?
  • Look! We just happened to have this big huge [REDACTED] lying around!

Also, watching the Batman vs. Superman trailer, this kept repeating in my head:

Flex Mentallo

But no, Superman must be grim and gritty! Batman must have that damn exo-suit! Aren’t we just simply tired of the simple politics of Dark Knight Returns by now? Can’t we move on, like Morrison did back in 1997 in JLA?

Hn. Clark.

The Not-So-Strange Death of Liberal England

Oh well, it only took 100 years for them to come back last time…

And Then There Was One

It’s quiet in the house tonight. I’ve had my family visiting for the past week, and the house has been a bustling hive of activity. Curtains have gone up, the door frames have been painted, fences mended, floors screwed down, power points installed front and back, signs put up, wasps eliminated. And there was also time to sit around the television as a family to watch thirteen hours of graphic violence. Oh, and I also saw Sleater-Kinney, who are indeed still awesome, and Turn Me On is every bit as vital when I heard it in 1997.

I am old

To add to the already full house, I then invited lots of friends to come and visit for a picnic (which was moved indoors after the to-be-expected rains were forecast). That involved a day of cooking, four different desserts and three more people sleeping in the house (though, granted, two of them were smaller people. Smaller people who were impressively interested in Transformers. More of this!).

Now, my family are somewhere past New England, ready to enter Canada and head across Greenland over the Atlantic. To home and waiting cats. The house is quieter, but feels as if my family has finally had more involvement with it, leaving their stamp on almost every room in the house, improving things and leaving me with a list of things to improve. Dad was even impressed with my choice of electric screwdriver.

And so, quiet evening on a long sunny Sunday evening. Knowing that I have great friends and family, even if there’s nobody here right now, there will be somebody dropping by before too long has passed.

(also, if you happen to be that person: I HAVE SO MUCH LEFTOVER FOOD AND I WILL MAKE YOU TAKE SOME)

crickets

crickets

Pun Leaderboards!

It’s been a long week, mostly for reasons I can’t talk about yet (nothing too exciting, just unexpectedly working on something that is currently embargoed), but I have at least just come back from a relaxing weekend where we all died horribly at the hands of an invincible space-cruiser.

(yes, we played Space Cadets again. We’re getting better! This time we got two crystals and died from a warp core breach! Next time, I may even introduce the Jump Drive station to the team…)

I also discovered that Columbia has a better deal on Callebaut chocolate than I’ve ever been able to find on-line, made a birthday cake, used a pizza peel and a steel plate to make pizza, almost drowned in a bowl of bottomless ramen (so good!), and was called upon to explain various bits of Britishness present in Miranda. A good time spent with friends, who just happen to be two interstates away. But! I’ll be back there next week when I go to POSSCON! My car is getting a work-out…

(though the main part of that was explaining who Raymond Blanc is. Oh, and Gary Barlow)

Oh yes, and I inadvertently started a pun competition and a pun leaderboard. I blame the eggsuberance of Easter for that (another point for me!). And then I left, heading back to NC, thinking nothing of the poor people that have to suffer the puns in my wake…

Moving On

Life as a consultant means you have to get used to saying goodbye to co-workers fairly regularly. And you can never be entirely sure how long that’ll be; I’ve worked on a few contracts in the past that have only lasted a few hours (pre-launch optimizations and the like).

But the last contract has lasted over a year, and I’m sad to say goodbye to everybody up in Boston, though it has been great to leave after finishing a major project which will touch most of the teams working at mumble mumble in the upcoming months. Which is something.

I will not miss the flatbreads at the Courtyard Marriott, though. Or the piles of snow all around (still several feet high in places, and it’s almost April!).

The iPhone saga from last week has been resolved with a quick trip to the Apple Store and a shiny new phone. Hurrah!

Crack’d Mirror

Firstly, I can sense a lot more pizza in my life in the next few months. I cooked my first New York-style pizza on a steel slate on Friday, and despite deforming one half of the pie due to a lack of a peel, it was the best home-made pizza I’ve ever tasted and came close to the heavenly taste of IP3. There shall be more.

In less happy news, my iPhone now sports a rather shattered look, having fallen foul of a kerb just outside a Walgreens in Durham. Looks like I will have to pay Apple the $80-or-so stupidity tax in the next week or so to get it fixed. It clouded what was at that point a rather nice Saturday. So if you saw a sad panda in the shape of a British person sitting alone outside Pelican’s eating a sno-cone slowly in the afternoon sun today…now you know the rest of the story.

Finally, after a brief foray into 60s British films on YouTube this evening, I’m fairly sure that around 73% of all British films made from 1964-1974 are incredibly suspect. I knew about I Start Counting, but seriously, what possessed people to make Twinky? couldn’t even make it through the title sequence…

Profiling Ruby

Last Tuesday, I gave a talk at the Triangle.rb meetup on profiling Ruby applications. Would you like some slides? I thought so!1


  1. A few slides were added from the talk, in particular adding an extra flamegraph example. There may be more to come on this front… [return]

Snow Week 2 - Morse Edition. And Brutalism

Westgate Car Park

It’s pretty ugly and I don’t think too many people will be sorry to see it go – as far as I know there have certainly never been any requests for its preservation.

Well, thanks, Oxford Mail. Yes, I know, the car park isn’t all that important in the scheme of things, but you wouldn’t say that if the city was about to pull down one of the Elizabethan buildings. Or a Victorian one. Or even some of the inter-war houses. Of course, the council had already ripped the heart of the park out over a decade ago when it took out the ‘golf-ball’ (where the security team watched over all in a rather Prisoner-esque fashion) and covered up the gaps in an amateurish manner. I’ll miss Westgate, even though I realize that not every building can be saved and a city cannot be trapped in amber forever.

In other news, I have spent the week at home again. Snow and plummeting temperatures once more. As cabin fever set in, I took solace in Inspector Morse and my new friend Bob:

Bob

Bob says hello and that he’d like to meet you all. One by one until he’s finished. Forever.

Tomorrow, back in the office. Providing it’s still out there. Is there anything outside the house? I am suspicious.