Are We Festive?

(kafka, redis, d3, docker, going too far? me?)

Are We Festive?

The idea came from me thinking how I was going to wind my sister up this year. I like amping up my FESTIVENESS every year, and I’ve only got worse since I left the UK. Now, I’ll be back in BIcester in a few weeks, and I have a few extra surprises to come, but I thought I would construct something that would show Bonnie that the world is quite festive and she should let go of her inner grinch. And a website would be just the thing!

Behind the scenes, it’s pretty simple. I’m hooking into the Twitter Streaming API, tracking every mention of ‘Christmas’ and sending a few bits of information from the tweet into a Kafka cluster. After that, I have a few scripts that run the tweets through a sentiment analyzer, classify them according to date, and produce daily and total sentiment averages. Everything then gets sent to Redis to be pulled out by a simple Sinatra server for the webpage.

Said webpage is a smattering of D3.js to create the two gauges displaying the averages, along with a simple polling function to update the page with the latest information. Plus snowflakes!

It didn’t take too long to put together, which is good, as I’ve found myself pretty busy these past few weeks. Everything got assembled as Docker images, which meant developing and uploading to AWS was a breeze (no fun surprises between development and production!). Sadly, I didn’t have enough time for all my original plans, so I had to drop my idea of having a world map and populating it with regional data, allowing us to see the most festive country as Advent unfolds. But that, along with a re-write of the processing logic in either Storm or Samza, could be a project for next year!

Meanwhile, Bonnie’s response after being sent to the website:

You need help.

She might be right.

blog comments powered by Disqus