StructType(Array(StructField('wibble', StringType, true)))

This is some great advice. In entirely unrelated news, as it was doing the rounds on Twitter, I was into day 3 (or 4?) of delving deep inside pulsar-spark. What I have ended up with is a Lovecraftian horror that dynamically intercepts Spark’s schema creation, adds a new StructType field, and in the deserialization pass, dumps the entire content of a Pulsar message into that field in case something goes wrong in the other fields1. After a week of 12-hour days, I can only reiterate that computers were a mistake.

As you might imagine, spending all that time working this week, I haven’t really had much of a chance to do much else. Other than spending Saturday night reading until 2am finally finishing the Horus Heresy sequence of books2 and through a variety of circumstances, ending up owning a 50” 4K TV that currently sits a little awkwardly in my bedroom. Although at least on that last one, the walk through Wal-Mart 70”-and-up range means you come out feeling like you could have been more ridiculous but were strong? Seriously, at some point, the TVs just become the size of a wall.

And no updates to But This Was A Fantasy either. Next week, though - three day weekend! So maybe, if you’re lucky, you might get something.

I am now going to curl into a corner until Monday comes.

  1. Which it does. [return]
  2. A special shoutout to Horus Heresy: Titandeath, which is Games Workshop essentially saying: “Hear us out: what about space mecha pilot lesbians? How triggered are you now, 4chan?” [return]

Moderna Part One

I have joined the ranks of the vaccinated! First jab, anyhow. I go back for the second Moderna shot during my birthday weekend, so by mid-May, I guess, my body will be at full mRNA power. The summer awaits!

(But I am tired and will be spending the day in bed. And Sunday too, as it happens)

While I prefer the British approach of the NHS just telling you when you’re going to get a vaccination…I guess I am lucky to be in a position where I can spend two days refreshing a browser window every five minutes until a slot becomes available. In a supermarket no less.

But still, remain indoors. No matter how sunny it looks outside this weekend. My hope is that we’ll finish the DIY projects that have popped into my head over Easter, and by the time the house can receive visitors again, it’ll look a little bit more like a home, and less like a show house that has two bedrooms jammed full of stuff everywhere. Look, I’ve only lived here for three years. You have to take your time with these things.

(also, Helvetica has started attacking one of the foam mattresses in one of the bedrooms, so that needs to be stopped before she builds a nest of foam pieces on the carpet. She’s an odd cat)

The “oh, yes” prize of the week goes to United Airlines, who this week sent me an email reminding me that the travel voucher we got in lieu of our trip to Japan last year is valid for another 24 months (I had completely forgotten about it, to be honest). Oh, wouldn’t it be nice to use that someday?

Next week, I promise I’ll get some updates done for the Curtis search engine. Maybe I’ll encode The Living Dead

Furnaces & Driveways

Okay, if the fates could please let up after my optimistic comment a couple of weeks ago, that’d be grand. In the past week, part of my driveway surround has collapsed and the furnace died. The poor HVAC engineer handed over the quote and let me stare at the zeros until I sheepishly said “it’ll be just as much to replace it, won’t it?” We decided that it wasn’t a good idea to do all that patching up of a 20-year-old furnace when for $200 more, I could get a brand-new, much-more-efficient furnace that doesn’t have holes in its heat-exchanger pushing out carbon monoxide1. Which I now have, thanks to a very prompt installation, but oh my, an expensive week.

Also, I am not loving this weekend’s time change, as with it, somebody using the main road as a drag strip at 3am, seemingly floodlights being left shining at my window, a very warm bedroom (hey, the new furnace works!), and a cat determined to scratch her way into a closed bedroom, I haven’t got a lot of sleep this weekend.

I still fit in my jeans, though. So that’s the high point of the week sorted.

  1. Don’t worry, I have a CO detector and it hadn’t pinged at all. [return]

Sorry, Could You Repeat That?

Obviously, my sense of optimism on display last week was leading me to a downfall. I forgot to centre myself appropriately.

My penance was to spend the week without being able to hear a lot out of the left side of my head. It made a very meeting-heavy week something of a challenge, I can tell you. As of right now, after a visit to the doctor, my hearing has been restored, but I’m now on more medication for the rest of the week to stave off an infection. It’s going great, and I’ve spent most of the week moping about and not doing anything. Which makes writing this weekly blog update a little trickier than usual.

In lieu of other exciting news, then, But This Was A Fantasy now has The Mayfair Set added to the encoded documentaries, so go to it for all your James Goldsmith and escalator needs. I’ve also changed the reranking algorithm to throw out anything similar within twenty seconds (up from five), and the front-end now also displays the TX date for each episode.

One thing I’ve noticed whilst testing CLIP out on new encodes is that it is very, very keen on matching to text. This is actually quite helpful within the documentary structure, because if you search for “Geoffrey Hinton”, it’ll match on the introductory text that comes with his appearance, and thus scenes with him in without the text will probably be similar enough that they’ll also get recommended. Unfortunately, it does mean that one of the smartest models around is quite stupid when it comes to mislabelled images.

Even a ResNet-16 probably wouldn’t be fooled by that one. Anyway, more encodes to come — I am very fond of 830,000,000 – Nick Leeson and the Fall of the House of Barings, and given that it now has a shiny new BBC iPlayer version instead of the VHS copy that’s been passed around the Web for years, it’ll be nice to get it up.

It's Only Make Believe

I’m not sure if it’s the snow melting, that we’ve reached 2m+ vaccinations a day over here, the approval of the one-shot J&J vaccine, the arrival of two servings of ginger cake and three bags of Mini Eggs, a rather decent week at work, or more likely a combination of all of these…but I’m finding myself a little more optimistic this weekend.

It was also a great week for old television. Not only did I discover that somebody late last year uploaded the BBC’s 1975 drama State of Emergency, I also got hold of my illicit DVD-R containing Dennis Potter’s Angels Are So Few and Only Make Believe. I do wonder just where the originals came from; there’s timecodes burnt into both plays, and a physical VT countdown clock at the start of Only Make Believe, almost as if the video came direct from the BBC archive (there’s no evidence that they’re off-airs — it’d probably be a bit obvious if the were considering both plays are from the start of the 70s). Anyway, that mystery aside, who doesn’t love a play with an ‘angel’ and a frustrated housewife and another play about a frustrated writer writing the first play and trying very unsuccessfully to seduce his typist? With the actor playing not-Potter being the same actor who played not-Potter twice already in the earlier Wednesday Play strand? Now that’s what I call a Saturday night.

Finally for this week - a couple of updates to But This Was A Fantasy. Firstly, I’ve added all the episodes of Pandora’s Box, so you can now search for “RAND Corporation, Santa Monica” and see where I ended up sometime in October 2011 after walking all afternoon. Secondly, I added the re-ranking function I talked about last week, so it should filter out any images that occur within ten seconds of the higher-ranked images. More variety! The BBC has made a few more series available on iPlayer, so you’ll probably see some more series added in March (The Mayfair Set for sure).

But This Was A Fantasy — An Adam Curtis Search Engine

I built a new thing.

Presenting: But This Was A Fantasy: An Adam Curtis Search Engine

Yes, given any natural language query entered into its shiny, Helvetica-accented search box, and The Ghost In The Machine & The Machine In The Ghost will respond with five images that it thinks are most appropriate.

How does it work? It’s a mixture of OpenAI’s CLIP model and Facebook’s FAISS vector similarity library. I take every second of footage, encode it using CLIP and store the normalized vectors into FAISS. For searching, it encodes the text, again using CLIP, and computes the cosine similarity between the text vector and all the encoded image vectors, returning the top 5 results (which I then check against a SQLite database to pull out metadata such as the episode title, timecode, and the appropriate image URI). Basically, “tell me what image is closest to this bit of text?”

There’s not much more to it, except for some embarrassing Javascript which was mainly me spending the week doing my first ‘major’ web development since working on the Call of Duty registration system back in 2012. Apparently there are magic things like the Fetch API now instead of XMLHttpRequest! No longer do you need to use jQuery and IE polyfills!

(please, JS devs, do not actually look at the code; there’s a couple of laughable bits in there which are me falling back to “yes, I’m sure there’s a proper way to do this, but I am bored and I’ll just throw in this half-remembered hack that gets the job done)

So it’s a FAISS index file, a SQLite database, a FastAPI Python server, one page of HTML and a lot of images, all bundled up into a single stateless Docker container and running on Google Cloud Run. Look at me embracing the current zeitgeist (in fairness, I am pleased with the Knative stuff and I’m already pondering how I can use it with work’s Kubernetes infrastructure to make our lives easier). And aside from having to adjust the memory limits to make the container actually run and some long-ish startup times, it all seems to work pretty seamlessly.

I do have some improvements in mind, if people are interested.

  • Obviously, add more series. The BBC have helped here quite a bit, as they’re put up a bunch of his older series up on iPlayer, meaning I can get high-quality videos instead of weird YouTube rips. I already have an index that includes all of Pandora’s Box.

  • If you play around with queries, you’ll notice that the search engine really likes returning images that are seconds apart from each other. This makes sense, as if the one second of an interview is a high match for the query, then the next second is likely to be as well. As is the next second. But it does make the results a little boring. I’ve written a reranking function that instead gets 10 results from the FAISS index and strips out those that have close timecodes to other images that come from the same episode. So I’m thinking of adding that as an option. Also, it might be handy to filter to only include results from a specific series, just in case you don’t want your search aimed at The Mayfair Set to be sullied with anything else.

  • Finally, I want to experiment with the different types of FAISS clustering so I can get the best use of memory, CPU, and accuracy out of the engine as I keep adding vectors to the index. There’s endless tinkering available here — I managed to take an episode down from taking up 10Mb to less than 750Kb without much trouble, but the accuracy of the results wasn’t great compared to the original. So I’ll need to come up with some metrics for (automatically) testing out the options.

  • Map ‘rainy fascist island’ directly to British-related images.

  • I’m not above doing a Perry & Croft or Morse, or even Cracker (“search for the odeon on the oxford road” —> “that’s fine by me!”) version if people are twisted enough to want those. Give me enough Plays For Today and we’ll get that going too.

Amusingly, this is pretty much directly related to my day job at the moment. Anyway, enjoy and let me know if you find anything fun!

Go Away Snow

Two weeks in and I can definitely say that I’m over the snow. Actually, I’m fairly sure I was over it a week ago, but two more snowfalls have seen more shovelling and having to jumpstart my poor cold car, so I really just want the temperature here to go above freezing for a few days and take all the snow away. Fewer flying sheets of ice on the roads, please!

(yes, yes, it’s pretty, and I don’t have it even a tiny bit as bad as colleagues in Texas. But I want it to go away! Maybe my car will start again)

Otherwise, it Has Been A Week. But it has gone a little quicker due to a project that I’ll be showing the first iteration of tomorrow morning! (It’s not that exciting, but prepare yourself for the first technical-ish post in a while!)

Finally, what do you do with leftover gianduja that you made for Shrove Tuesday? That’s right, NUTELLA DIGESTIVES.

View this post on Instagram

A post shared by Ian Pointer (@carsondial)

Can't Get The Generation That Bought More Shoes Out Of My Head

In the South of Britain, A Man Made A Website

Well, then. I’ll admit that when the music kicked in at the end of Part Four, my late-night reaction was “wait, is that the firework? OMG IT IS!”. And then, the next day, I found out that the BBC actually vetoed him calling the series You Are The Generation That bought More Shoes And You Get What You Deserve (apparently for being too long). I feel like my old, old website has finally been vindicated.

As for the series itself? As usual with Curtis, about 50% bobbins, 50% interesting. But probably his best since The Mayfair Set, I’d say. And so much of it! Even longer than the total runtime of Pandora’s Box (by about two extra hours (!)). I also like how so much time has passed that even his old interviews have now been turned into archive footage for Curtis to recycle1.

I have to applaud the use of Gell-Mann in the final episode; I can’t help but think that’s a slight nod to the Gell-Mann Amnesia effect, which was in full flight in the second half of Part Six when Curtis got the post-2008 and post-2016 timelines wrong and a rather misleading segment on DeepDream.

Still, if you fancy a fun and frustrating eight hours on a tour of the world from the 19th century up to the present day encompassing George Boole and Tupac Shakur, you can’t go too wrong with Can’t Get You Out Of My Head2. Given the focus on China, I was sad that it didn’t also dig in a little on Thiel and Nick Land. Chuck in a bit of Burial on there as a cheeky wink to Mark Fisher, and you’d be golden3.

One thing I would like to see one day (could somebody chuck some money to Robin Carmody to get this done, maybe?) is a proper look at how odd Britain was in the late 60s and the 70s? I’m talking about things like Lord Lucan, just-what-those-tanks-were-doing-on-the-Heathrow-runway, The Owl Service, the final days of Harold Wilson, GB75, Dennis Potter, Lieutenant Pigeon, the Yorkshire Ripper, the formation of the IEA, the summer of 1976, Jimmy Savile, Nigel Kneale, The Fall and Rise of Reginald Perrin, Derek Jarman, Jeremy Thorpe, Play For Today, Play Away, internment, decimalisation, Whatever Happened To The Likely Lads?, the Three Day Week, wife-swapping, and Arthur Scargill. More bits like the Whicker documentary in Part One!

  1. Lots of The Living Dead for a start. I was mildly annoyed that he didn’t return to RAF founder Horst Mahler, given the his turn to Holocaust denial since that interview [return]
  2. Two leaders conspicuous by their almost total absence from the narrative? Reagan & Thatcher. The lack of the latter (aside from a small archive clip) was especially glaring, given that Curtis made the ‘telling stories is powerful’ pitch in The Attic 25 years ago. But it would have also contradicted his pitch here somewhat… [return]
  3. Also, the lack of Carter The Unstoppable Sex Machine during the multiple mentions of Rachman was a crime. A crime! [return]

Sunday Evening Sad

Well, it finally happened. I’ve slipped into self-parody, and I watched the first episode of Last Of The Summer Wine this week. And you know…it wasn’t bad? Including a few gags that you’d be a little surprised to find going at 20:30 today, let alone in 1973.

(I also made it through two episodes of The Owl Service, which I liked…but dear God, some of the costume choices must have been questionable for a children’s show in 1969, surely? Alison doesn’t wear trousers or a skirt for the entire length of episode one. Just a little awkward)

Continuing a theme, though, I would like to register a complaint about my situation in this country. Why is American cling film so bad? It doesn’t stay in the box, the the cutters don’t really cut…and it doesn’t stick to anything except itself. It somehow manages to fail at everything. And this isn’t just me fondly remembering British cling film with a ‘halcyon days’ filter, as I have a roll of it here too — and it just works as you’d expect. Except I have to ration it as I won’t be getting extra supplies any time soon. I just don’t understand how the equivalent is just so bad here, across all types and brands.

I have watched three men in a tin boat and I’m complaining about cling film. Yep, I’m definitely in my 40s now. Also, we spent the weekend prepping for the cold days ahead and DIY. Ah, middle age…

Oh, but I did make Nutella Aero! That counts for something, right?

View this post on Instagram

A post shared by Ian Pointer (@carsondial)

All The Books

There’s absolutely no way I can keep this pace up, but as we come to the end of January, it turns out that I’ve read 81 books in the past 31 days (and probably around 10 deep learning papers on top of that). I guess the Winter nights have seen me head back to the bookshelves (as I write this, the snow is coming down in big clumps). I would like to put something together like for what books I get through during 2021, but you can’t get an API key for Goodreads anymore. So I’ll just keep a list instead.

(and yes, I’m already trying to work out where the next set of bookcases will go. My end goal is something like Eco’s, naturally.)

We finished It’s A Sin this week! Cheery! Although I did enjoy the parts where I had to pause and explain Isla St. Clair, Larry Grayson, the Generation Game, and other bits and pieces of 80s Britain along the way. And although I knew it was coming, seeing this advert again brought back childhood terrors:

(One thing you can say about Britain is that we used to know how to make Public Information Films that would instil the Fear of God in children across the land)

Anyway, good series; I feel that knowing that RTD planned it as an eight-part series hurts it a little — things feel a little too compressed at times, and I think the dropped plan for the final episode (which would have brought Jill into the present day) would have rounded things off better than cutting the story in the early 90s. Or maybe I just wanted it to end on a bit more of a happier note.

And I think that’s it for this week. Next week should see at least one experiment up on the blog. Maybe two if I can summon up a little more effort to get motivated than I normally do these days. A dark Winter indeed…

  1. When I started this entry last night, the figure was at 7. Then I stayed up to 2am reading Piranesi from start to finish. I may be slightly regretting not splitting it out over two days this morning. [return]