Planes, Trains, and Automobiles — DC Edition

The first inkling that this weekend was not going to go entirely to plan came on Tuesday. American Airlines pushed an update to my phone that evening saying ‘well…er…we think there’s going to be bad weather on Friday evening. Fancy rebooking?’ Like a fool, I ignored it, thinking that it’d all work out, and besides, if I was a little late, it’d be fine. Totally fine.

By the time I got to the airport on Friday (after sending off all evidence that I’m a US citizen in order to get a passport), the flight had already been delayed until 20:30. But the plane was in the air, so I’d still get to DC around 22:00. Not great, but not the end of the world.

We started boarding around 21:00. We got to the runway at 21:30. number one for take-off. Okay, so maybe I wouldn’t be meeting Richard tonight, but I’d get to my hotel no problem.

At which point, obviously, the plane developed a mechanical fault and returned to the gate. Mind you, it’s better to develop a faulty engine on the runway than having one 6,000 metres up. Anyhow, back to the airport we went, and a sinking feeling descended. Especially as the people in front of me in the rebooking line were getting more and more irate about being offered flights for tomorrow evening. As Richard would be flying home on Sunday, it wasn’t really good for me either.

The day was saved by a combination of a ticket agent happy to have somebody not gearing up for a fight, and Tammy, offering a helpful suggestion - there was a Megabus service that would leave Durham tonight and arrive in DC in the morning. Hurrah!

Just one tiny little catch: the bus would leave at 03:30.

Having confirmed with the ticket agent that they’d hold my return flight open for me if I didn’t rebook my outgoing flight, I headed home, had a shower, drank multiple cups of tea, and started working out how exactly I was going to get to the centre of Durham at 3am. I didn’t really come up with a good solution aside from maybe there will still be some Lyft cars about at 3am?¹. Thankfully, the downtown bar scene still has stragglers at that point on a Friday, and so there I was, taking pictures of the Mutual Life building at 3am and hopping on a double-decker bus to DC. By hook or by crook, I was going to get there.

We’ll pause now to list some of history’s greatest monsters:

• The people who sit on the aisle seat but leave the window seat clear but inaccessible
• The people that have an hour-long call about banking processing charges where they repeat that information approximately every five minutes into the conversation
• And the people that get on at Richmond, sit next to you and start playing music on their phone. Headphones still exist, everybody².

Ahem. Anyway, I did get some sleep, and even arrived in DC 15 minutes than scheduled. A full day of tourism approached!

It’s easy to forget how big the Mall really is, and how impressive the monuments and the planning of central DC comes together. We spent twelve miles in the quest of rediscovering that. But it was fun to play tourist and show a friend around. I did have some impressive bad luck on that front - ending up at a Mexican chain restaurant for lunch after walking to a series of either closed or non-seating places.

I made a better choice for dinner (my first Ethiopian in a few years), but my choice of going to the nearest whiskey bar to save on walking turned out to have two slight issues: 1) its whiskey selection was not exactly stellar, and 2) the research I had done online neglected to mention that it was a gay bar. Which is fine…but I don’t think either of us expected to be chatted up in the evening, or to have somewhat risqué conversations about Bicester Village.

After finishing our drinks there, we wandered and wandered, eventually ending up here (look, it’s a local chain, okay?) for the rest of the evening.

The trip back was much easier. The plane worked, I landed at RDU, and I’m now laid up looking at developing blisters. I don’t have to go outside again until Friday…by then they should be going down some…until then, ibuprofen it is!

Breaking Neural Networks, Part 1 — Turning A Frog Into A Cat

We’ve all seen the hype. Deep Learning techniques can recognize human faces in a crowd, identify seals from orbit, and classify insects from your phone! SkyNet awaits!

Well, not today, everybody. Because today is the day we fight back. But we won’t be sending back John Connor to save the day. Oh no, we’ll be using maths.

Okay, so first, we need a neural network to attack. Today, we’re not going to be sophisticated - we’re simply going to take an architecture from one of the PyTorch tutorials. It’s a very shallow ConvNet for classifying images from the CIFAR-10 dataset, but the principles here scale up to something like Inception. We train the model, again just using the training code in the tutorial, and we begin.

What is this, oh clever neural network?

Model says: Frog

(CIFAR-10 images are only 32x32 pixels, so yes, it’s a touch blocky)

What we’re going to do is change our picture of a frog just enough that the neural network gets confused and thinks it’s something else, even though we can still recognize that it’s clearly a frog, thus proving man’s continued domination over machine.

To do this, we’re going to use a method of attack called the fast gradient sign method, which was detailed back in 2014 in this paper by Ian Goodfellow , Jonathon Shlens & Christian Szegedy.

The idea is that we take the image we want to mis-classify and run it through the model as normal, which gives us an output tensor. Normally for predictions, we’d look to see which of the tensor’s values was the highest and use that as the index into our classes. But this time, we’re going to pretend that we’re training the network again and backpropagate that result back through the model, giving us the gradient changes of the model with respect to the original input (in this case, our picture of a frog).

Having done that, we create a new tensor that looks at these gradients and replaces an entry with +1 if the gradient is positive and -1 if the gradient is negative. That gives us the direction of travel that this image is pushing the model. We then multiply by a scalar, epsilion to produce our malicious mask, which we then add to the original image, creating an adversarial example.

Here’s a simple PyTorch method that returns the fast gradient sign tensors for a input batch when supplied with the batch’s labels, plus the model and the loss function used to evaluate the model:

    def fgsm(input_tensor, labels, epsilon=0.02, loss_function, model):
        outputs = model(input_tensor)
        loss = loss_function(outputs, labels)
        loss.backward(retain_variables=True)
        fsgm = torch.sign(inputs.grad) * epsilon
        return fgsm

(epsilon is normally found via experimentation - 0.02 works good for this model, but you could also use something like grid search to find the value that turns a frog into a ship)

Running this function on our frog and our model, we get a mask that looks like this:

Adding that mask to the frog results in:

Clearly, still a frog. What does our model say?

    model.predict() -> "cat"

Hurrah! We have defeated the computer! Victory!

Eagle-eyed readers will have noticed that in order to produce an image that fools the classifier, we need to know a lot about the model being used. We need the explicit outputs so we can run backpropagation to get our gradients, and we need to know the actual loss function the model uses to make this work. This is fine for this example, because we know everything about the model, but if we wanted to fool a classifier where we don’t have access to any of these internals, we’re stuck. Right?

Stand by for Part 2 soon, where we discover how we can extend this approach to defeat classifiers where all we get is ‘this is a frog’. We shall defeat the machines!

Two Feet High And Chomping

A post shared by Ian Pointer (@carsondial) on Jul 22, 2017 at 5:44pm PDT

And with that, I’m pretty much coming to the end of my recent buying spree of Transformers. Trypticon was one of the toys that never made it to UK shores, so I was always going to be a sucker for Hasbro’s current strategy of ‘let’s do all of G1, but this time with knees’. So I now have a two-foot dinosaur to go with my two-foot robot…but with no idea where I’m going to put them (although I guess I can delay a permanent decision there for a while).

Next week, more adult pursuits: getting my first US passport. Exciting times!

Rusted Ruby

I gave a talk this week! If you’re interested in a 10,000m look at Rust and then a delve into Helix, then here’s your slides!

The talk suffered a bit from my usual deer-in-headlights presenting style and that I lost a week of preparation due to wisdom teeth shenanigans, but hey, I implemented a 3x speedup of Ruby’s Pathname.absolute? in less than ten minutes, which isn’t too bad…

Going to a busy six weeks coming up - as of yesterday, I have trips to DC, Cincinnati, and of course Bicester and London before the end of August. And then…and then, things get even more fraught as I begin my 2018 masterplan. Which is much less impressive than it sounds, but will involve quite a bit of upheaval. But all for the better in the end!

Hopefully, I will sleep a little more this upcoming week, too. And fewer exam stress dreams set back in Manchester. It’s been 17 years…

Finally...Tea!

I can drink hot and carbonated drinks again. Which is good, because I seem to have bought all of the Diet Coke in Durham during the past two weeks of Target sales. I may have miscalculated how much even I can drink, but you can’t beat 4 cases of 12 cans for $8.88, can you?

Other than that…it has been a quiet week. I mean, really quiet. The most exciting thing of the has likely been the discovery of 1980’s Fox. Come back next week for more exciting updates!

The Colour Problem - Neural Networks, the BBC, and the Rag-and-bone Man

tl;dr - a fully convolutional neural network trained to colourize surviving black-and-white recordings of colour Steptoe and Son episodes.

One of the most infamous parts of the BBC’s history was its practice of wiping old episodes of TV programmes throughout the years to re-use videotape. The most famous of these is, of course, Doctor Who, but it was far from the only programme to suffer in this way. Many episodes of The Likely Lads, Dad’s Army, and Steptoe And Son have been lost.

Every so often, an off-air recording is found. Recorded at the time of broadcast, these can be a wonderful way of plugging the archive gaps (who said that piracy is always a crime, eh?). If the BBC is lucky, then a full colour episode is recovered (if it was broadcast in colour). More often for the older shows, however, it’s likely that the off-air recording is in black and white. Even here, though, the BBC and the Restoration Team has been able to work magic. If a BW recording has colour artifacts and noise in the signal (known as chromadots), then the colour can be restored to the image (this was performed on an episode of Dad’s Army, ‘Room At The Bottom’).

If we don’t have the chromadots, we’re out of luck. Or are we? A couple of months ago, I saw Jeremy Howard showing off a super-resolution neural net. To his surprise, one of his test pictures didn’t just come out larger; the network had corrected the colour balance in the image as well. A week later, I was reading a comedy forum which offhandedly joked about the irony of ‘The Colour Problem’ being an episode of Steptoe and Son that only existed in a b/w recording…and I had an idea.

Bring out your data!

Most image-related neural networks are trained on large datasets, e.g. ImageNet, or CoCo. I could have chosen to take a pre-trained network like VGG or Inception and adapted it to my own needs. But…after all, the show was a classic 60s/70s BBC sitcom production - repeated uses of sets, outside 16mm film, inside video, etc. So I wondered: would it make sense to train a neural network on existing colour episodes and then get it to colourize based on what it had learnt from them?¹

All I needed was access to the colour episodes and ‘The Colour Problem’ itself. In an ideal world, at this point I would have pulled out DVDs or Blu-Rays and taken high-quality images from those. As I’m not exactly a fan of the show…I don’t have any of those. But what I did have was a bunch of YouTube links, a downloader app, and ffmpeg. It wasn’t going to be perfect, but it’d do for now.

In order to train the network, what I did was to produce a series of stills from each colour episode in two formats - colour and b/w. The networks I created would train by altering the b/w images and using the colour stills as ‘labels’ to compare against and update the network as required.

For those of you interested, here’s the two ffmpeg commands that did this:

    ffmpeg -i 08_07.mp4 -vf scale=320:240 train/08_07_%06d.jpg
    ffmpeg -i 08_07.mp4  -vf format=gray traingray/08_07_%06d.png

I was now armed with 650,000 still images of the show. I never thought my work would end up with me having over half-a-million JPGs of Steptoe and Son on my computer, but here we are.

But First on BBC 1 Tonight

Having got hold of all that data, I then took a random sample of 5000 images from the 600,000. Why? Because it can be useful to work on a sample of the dataset before launching into the whole lot.

As training takes a lot less time than on the entire dataset, not only does this allow you to spot mistakes much quicker than if you were training on everything, but it can be great for getting a ‘feel’ for the data and what architectures might or might not be useful.

FSRCNN

Normally, if I was playing with a dataset for the first time, I’d likely start with a couple of fully-connected layers - but in this case, I knew I wanted to start with something like the super-resolution architecture I had seen a few weeks ago. I had a quick Google and found [FSRCNN]( Accelerating the Super-Resolution Convolutional Neural Network), a fully-convolutional network architecture designed for scaling up images.

What happens in FSRCNN is that the image is exposed to a series of convolutional layers which reduces the image’s complexity down to a much smaller size, whereupon another set of convolutional layers operate on that smaller data. Finally, everything goes through de-convolutional layers to scale the image back up and then to the required (larger!) size.

Here’s a look at the architecture, visualized with Keras’s SVG model rendering:

A Citizen With 4 Fewer Teeth

I no longer have a green card! Or anything that identifies me as a valid citizen rather than a large, watermarked piece of paper. Which is likely worse…but better, once I get the new passport sorted out. Anyhow, the citizenship ceremony was fairly straightforward, except that our current President still hasn’t got around to recording a welcome message for new citizens, and a complete absence of him from all printed material. Not that most of us minded when we realized.

After a weekend of boardgames, Doctor Who, buying 8 cases of Diet Coke, and a longer-than-expected-dinner-because-reasons, it was Monday and time to celebrate being an American in style: getting my wisdom teeth removed. I have always been hesitant to have them out, due to one of them being apparently close to one of my jaw nerves. But the dentist here seemed very keen on yanking them out, and well, I was getting tired of the infections they kept bringing. I was informed that I have the smallest mouth she’s ever seen…but I was also one of the nicest patients she’s had. So swings and roundabouts there (everybody seems to agree on the small mouth thing…BUT NOBODY EVER SAID THAT TO ME BEFORE! But fine. Sure.).

Thankfully, it was fairly quick and easy to yank them out, but as everybody thought it would be a bad idea for me to be alone afterwards, Tammy drove me all the way back to Kentucky for the week. I spent the 8-hour car ride apologizing every five minutes, so I’m grateful that she didn’t throw me out in West Virginia.

Aside from running a fever on Wednesday and making the well-intentioned mistake of eating pizza yesterday, it hasn’t been too bad. Some pain, but not unbearable, not too much swelling, and the nerve ended up not being a problem. Hurrah! Though I am looking forward to being able to drink Diet Coke and tea again next week.

In the meantime, I am hopped up on ibuprofen, antibiotics, and percocet, whilst doing work and dying over and over on Zelda¹. Tomorrow, mostly recovered, I fly home to Durham. Home for now, anyhow…

Brian Cant

RIP.

As a point of comparison to American readers, I’d say that Cant was something akin to Mr. Rogers in terms of importance to younger viewers. But whereas, to this foreigner, Rogers comes across as the classic American staid gentleman next door, Cant and the Playaway/Play School set gave the impression that they spent their evenings debating Marxist tracts in their omnisexual polyamorous commune.¹ They were firmly pitched in the ‘now’…even if that ‘now’ seems a lost time and place for us, particularly in the light of recent events.

Brian Cant, star of BBC kids' shows Play School and Play Away, and narrator of Trumpton, has died aged 83. https://t.co/6HfAjMq2u0 pic.twitter.com/Q6VKSzZxA4

— BBC Four (@BBCFOUR) June 19, 2017

The other thing about Cant is that he was always there. We grew up with him on Playaway and Play School, but even after going to school, you’d see him somewhere when you were off sick and watching the schools programmes. Or when he was doing a rotation on Jackanory. Or during the Summer when repeats of Camberwick Green or Trumpton would fill in time on But First This… And then even later at university in the late 90s, where he did The Organ Gang shorts for This Morning With Richard Not Judy. A reassuring twinkling smile when you’re drinking a lemsip…with a hangover on a Sunday morning at the age of 28.²

You can’t claim that he was one of the Pythons, or up there with Spike Milligan, but he had that sparkle of safe anarchy that us British and children in general love:

You could never describe Cant as cool. And yet, a generation of us watched him whilst we were small and knew that’s what we wanted to be when we grew up. Even if we didn’t realize it until much later.

One More Day

My family got an extra night in the US after their plane failed to reboot properly. As of right now, they’re delayed another two hours on their second attempt. At this rate, they’ll be here for my citizenship ceremony on Friday. But hopefully they’ll get underway this evening. After all, the cats back home are getting hungry without their supply of treats…

The Absolute Boy

22:00 BST. When the Exit Poll fell.

Although I didn’t make it explicit last week for fear of jinxing it, the YouGov/Survation polls of last week didn’t just make me hope. Given the completely inept way the Tories ran their campaign, the other polls just seemed wrong - surely we wouldn’t give somebody a 100+ seat majority when they spent six weeks seemingly hiding from the press?

We did not.

And we laughed and laughed and laughed. The Tories achieving an amazing Pyrrhic victory, managing to lose a 25-point lead to a man that just two months ago looked like he was taking Labour to the point of destruction.

But they were wrong.

Along with Macron’s En Marche giving people a trouncing in the French elections, things might be looking up¹? Just maybe?

In less globally-important news, my citizenship interview went well, and I will become a US citizen on June 23rd. I will celebrate by having my wisdom teeth taken out on the following Monday. I know how to have a good time, y’know.

And, a good time was had this weekend - a full house with many friends, Tammy and I spending Sunday making cakes, ice creams, other pastry items, and then me abandoning her to cook all the chicken. But: so many people that even the extended table wasn’t enough for everybody. Pools and slip’n’slides as well!