Eyes Left, Eyes Right

Well, I guess it has been a while (Can’t Get You Out Of My Head was 2021, after all). It also looks like we’re back in firm Curtis territory versus the narration-free TraumaZone.

This means I really do have to get a move on to update But This Was A Fantasy with all the updates I’ve been planning for a while - mainly: using a vector backend that allows me to do proper metadata filtering instead of the current system hacking bits onto FAISS in an attempt to make filtering work. I’ll also take the opportunity to re-encode all the images from the documentaries with SigLIP2 to get better text-image similarities too.

Anyway, the hope is to get most of the changes and migrations underway next weekend so I’ll be good and ready to upload the new series very soon after broadcast. Let’s see if that really happens…

Otherwise, back from Vegas and celebrating Tammy’s graduation this weekend (Masters #2 for her!). Thank goodness for Memorial Day next week…

A Change In Scenery

A later blog this week, because as I write this, my computer is a couple of thousand miles away to the east. I, however, am in Las Vegas, where I have flown in to surprise my family while they’re on their trip. And they were somewhat surprised! So now, it’s very much a traditional Pointer holiday, at least until Monday afternoon when I fly back home. Also, a quick stop to an out-of-the-way liquor shop which appears to have the last stocks of last year’s Wild Turkey 70th Anniversary edition left in the country (seemingly a complete case of it, too).

The odd thing when talking about being in Vegas is that everybody expects me to spend the 48 hours on the Strip, but our family’s visits to the city have evolved to the point that we hardly ever go there when we’re in town. Which sometimes leads to us wandering around a very large set of fields in 35ºC heat trying to find food trucks, but it does let us see more of the place than a lot of visitors do.

(Some things are traditional though. I got propositioned twice by ‘adult entertainers’ in the first 24 hours of being here, and my family found a way to watch football matches in the morning)

Once again, I brought the iPad with a view of using some quiet time to hack on an embedding idea I have; and once again, it has fallen flat. I don’t know what’s at fault, but even with a keyboard and a mouse, Google Colab is pretty much unusable in Safari, locking up the screen on a simple small copy/paste and scrolling sometimes working, but most of the time being a janky mess that causes a reload of the tab randomly. So I guess, back to just scribbling on papers with the Pencil; which admittedly, it does do very well. But sometimes, I want to go beyond scribbling to playing with ideas, but I don’t want to lug my laptop around.

May????

Astounded at how fast the year has gone. Already May seems to be completely locked up (with some news this week, I think the only free weekend I have all month is Memorial Day weekend…which at least has a Monday off too. But busy busy busy otherwise.

And as I don’t have a lot else to talk about this week, welcome to Ian’s Round-Up of TV Things He’s Watched FROM THIS YEAR!

The White Lotus S3: I feel cheated that the central UNC/Duke tension was not resolved. Otherwise, a bit of a step down from S1 and S2, longer in length, but feeling twice as padded. Still, here for Carrie Coon rave scenes.

Andor S2 (so far): Not the greatest show that television has ever produced, as some would have you believe. But here for the Mon Mothma rave scenes. And it really is the best Star Wars thing in absolutely ages.

Doctor Who S2 (so far): Considering how S1 went off the rails, it’s probably still far too early to say (if Anita Dobson ends up being The Rani than RTD needs a quiet talking to), but a decent run of episodes, probably the best since the Capaldi Era, even if the politics of Lucky Day were pretty muddled…

When GERD ATTACKS!

Currently suffering through a particularly acute acid reflux flare that makes me not want to do pretty much anything. Hopefully the medication will help out a little in the days to come.

My GRPO adventures in April were…pretty much a bust. I’m trying to do something that’s a little more complicated than just trying to do multiple choice answers to the AIME dataset and it is failing miserably. Possibly because I’m using too small a model, but that was the main point of the experiment. Not everything works out, but it has been a bit of a dry spell lately. But I haven’t entirely given up yet…

Next up: May. Fun and games ahead!

Regrets At 46

They say you can’t make it this far without some regrets. Well, I’m here to tell you that this Easter, I have a big one. I really should have picked up that 1kg bag of mini eggs in Waitrose back in February. I had plenty of space in the suitcases for another bag!! A nice round 5kg of egg goodness would have been just the ticket!

Anyway, a quiet but nice birthday. I finally visited The Toy Department out in Fairfield, confirming that people will indeed spend far too much money on old Transformers, and Maeryn even bought me some Lego! Plus, I now have the complete DC One Million Omnibus, which I think pretty much rounds out my Morrison DC collection.

In parenting news, I have finally caved and bought a Fire tablet on sale to be a wifi-less device to help with potty times. Loaded with all the Hey Duggee and Bluey that you can fit on there, I expect…

Adventures!

I feel like there needs to be a banner or something when you start potty training. “WELCOME TO THE NEXT LEVEL”, perhaps. Still, some progress on the first day…

Also, I’m wondering if there should be a WHAMageddon-esque contest for “how long can you avoid looking at your retirement account while the line keeps going down?”

52 Quintillion Minus Six?

Quick round up!

• I’ve now seen Eno four more times, and my thoughts from back in January haven’t changed. I’d also add that watching the film multiple times across a number of days really does highlight that the ‘spine’ running through the film is just too constant for the gimmick to work. The first time you rewatch, after a couple of months, you think ‘fair enough, there were a couple of new things there’. But after the third showing in 24 hours, it’s more ‘for god’s sake, that bit with taking photos of the beetles is in every version???’ I want a 90 minute Cybernetic Eno! Or 60 minutes on James! (okay, I might not want that all that much, but it’d at least be an interesting difference). It probably works better if you just watch a version once a year or so.

• Meanwhile, there’s attacking sacred cows, and then there’s Marcello’s brave reassessment of Be Here Now. I can’t go that far, though I’ve always felt Noel always wanted to be more musically diverse than you’d think…but this only showed up here and there across the Oasis albums because he always bottled it. The only thing I can listen to from it these days is D’You Know What I Mean?, which I will still stan for, but the rest of it…just no.

• April will be “GRPO month!” at Snappish Towers. I have two main experiments to work one; one will not be a surprise…the other probably won’t either, but if I can get it to work, it will be an interesting new approach to the particular problem

• It looks like the two cities I’ve lived in while being in the US have another thing in common: malls called “Northgate” that have failed. Right now, the Cincinnati one has one single occupant of the food court and the anchor stores are basically Torrid (closing in a few days), and Hot Topic (which admittedly, looks like every other Hot Topic and may be the last tenant standing)

• Adventures in toddlerdom: whereupon Daddy, Maeryn, and Helvetica all make poor choices which ends up with a cat walking around with melted chocolate all over its fur. Most of the bad choices were mine, but I’m still looking side-eye at the supposedly intelligent cat that decided that was a great place to sleep…

Now We Are Two

Obviously, the big news of the week is that our tiny little dictator, I mean, lovely little Maeryn, is now 2 years old! We celebrated with cake¹, balloons, presents, and the all-important Sunday Roast. Well, she’s older now, so it’s time for her to really understand her heritage. Next week: Dennis Potter. Or maybe she just plays with her new Little People Barbie Dream House for the moment…

The Atlantic had an exposé this week about how Meta used LibGen to train Llama models, along with a little search bar to see if a book or author is present in LibGen (and thus likely the text(s) was used to train a bunch of LLMs). I realized late in the day that I would likely be in the database, and lo…I have 3 entries, including the German version of my PyTorch book. I am mostly fine with this, and I’m more amused that some of my writing has gone into training Meta’s LLM how to write PyTorch code, PyTorch being a Meta-owned project. Of course, it’s easy for me to say that, being both a wizened anti-copyright person that came of age during the Copyleft Wars of the 90s², and somebody that doesn’t make their main income by writing books. I can see exactly where others are coming from, but I also don’t want to restrict us to a world where only OpenAI and Anthropic has the money to build and research models because nobody else can afford the usage fees on Common Crawl.

(also, I note that the AI narration that The Atlantic sticks on the article was almost certainly powered by using copyrighted content too…so…you’know)

Gist

Okay, so this month we’re going to talk about something that doesn’t really work. You’ve seen all those amazing arXiv papers with their fancy new discoveries? Bah, that’s easy. The true apex of research is talking about the things that failed. So welcome to a litany of failure!

Anyway, this is an idea I’ve been wanting to try out for at least a year. I have entries in my notes file that go back to 2023, and really I just wanted to get it out of my head and using my co-worker, Claude Sonnet, to determine if there was any promise in the technique.

Welcome, then, to ‘Gist’, an attempt to improve the search relevance of chunking in vector search!

(if you’d like a précis of searching with vectors, head over here and then come back. The tl;dr: use a model to create vectors of your documents, and then make vectors out of a query and select the ‘closest’ document to the query vector)

Problems With Embeddings & Chunking

When you’re making an embedding of a document using a ‘standard’ embedding model, one of the issues you’ll run into is that most of these models have a context length of 512 tokens or about 400 words.¹ This has given rise to the cottage industry of ‘chunking’, where a document is split somehow into small chunks and each of them is vectorized. This way, your searches can dig deep into the documents and hopefully get really good results as opposed to just searching across the first 400 words.

However…

Let’s consider a dumb-but-common chunking method - splitting on sentences. Here’s a simple example:

The BT Tower is a grade II listed communications tower in Fitzrovia, London, England, owned by BT Group. It has also been known as the GPO Tower, the Post Office Tower, and the Telecom Tower. The main structure is 581 feet (177 m) high, with aerial rigging bringing the total height to 620 feet (189 m).

This gives us three chunks:

The BT Tower is a grade II listed communications tower in Fitzrovia, London, England, owned by BT Group.

It has also been known as the GPO Tower, the Post Office Tower, and the Telecom Tower.

The main structure is 581 feet (177 m) high, with aerial rigging bringing the total height to 620 feet (189 m).

If you were embedding these three chunks, and you were also embedding lots of other documents in the same way, it’s possible that your search is going to run into problems. If you have a query like post office tower height, you’re going to want that last chunk to score very highly. But that sentence, stripped from the rest of the paragraph, has no link to the concept of the tower whatsoever, and so neither is the embedding. Instead, what you’re likely to get is a response of all the chunks across your search index that mention height. Terrible!

The easiest fix to this, and one that would likely work well in this particular case is to split on paragraphs instead of sentences, so the embedding would have the context of the tower and the height in the same vector. But imagine a longer document, and you can see that you are likely to start missing context clues in your chunks, which will have a big impact on your search results.

But what to do…what to do?

The Idea

Admittedly, this is a pretty dumb idea, and it’s likely somebody has already done it before, but hear me out: what if the system could carry a CliffNotes version of the document with it for every chunk? That way, the search engine can be in page 23 of 113, but still have a general idea of what the chunk is talking about by relating it back to the notes. That should help boost the appropriate relevance when searching.

Turning that into an actual plan is even dumber: we get a LLM like Llama, Gemini, etc. to generate a 400 word summary of the document, taking advantage of their wide context windows to summarize the entire document. We then embed that, and here comes the magic!, we average this vector with every single chunk in the document.

It’s so stupid. And yet…compels me though…

A Brief PoC

So yes, I’ve had the idea rattling around in my head for a while, but never really enough time to sit down and do an evaluation. But then Claude 3.7 came out and I thought I’d use this as a chance to test that out and get this idea out of my brain.

Firstly, an evaluation dataset. I could have used MSMARCO, but I distrust it, knowing that a lot of the quality judgements in it are just plain wrong. Plus, every embedding model is trained on the MSMARCO data, so it’s not a fair test anymore (in my, and lots of others’ opinion). Instead, I downloaded 1000 random pages from Wikipedia and got Llama-3.3-70bn to generate possible search query terms for each page.

Now at this point, various IR people are yelling at me saying that that’s not fair either, as you can’t guarantee that the search terms I generate for one document are completely separate from any other document in the set…and that’s a good point, but this is not supposed to be a rigorous examination. It’s “does this even make sense to pursue?”

Anyhow, I now have documents, queries, and a mapping of what query goes with what document. Next up, we create a script that creates a summary of the documents, again using Llama-3.3-70bn², breaks the documents up into paragraph-level chunks, and then produces two embeddings per chunk, one just being the chunk itself, and the other being the chunk embedding averaged with the summary embedding.

Having got all that sorted, finally we embed the queries, identify the top-scoring chunks (and importantly, their document id), and write that out as a series of qrels run files, which we then use ranx to rank with the answer set and give us a nice set of summary tables and plots. (This is the “draw the rest of the owl” part)

Results

In my best Peter Snow voice, this is just a bit of fun. I used Snowflake’s arctic-embed-xs model for embedding, as it’s very small, quick, and capable, and tested Gist in a paragraph splitting scenario. Have some tables and graphs. What we’re tracking here is NDCG, which scores between 0 and 1, putting higher weight to the correct documents appearing at their correct rank towards the start of the ranking list.

Paragraph Chunking & Averaged Summary Chunks

Here’s the data formatted as a markdown table:

A Wild Sava Approaches

Having seen the big jump in NDCG scores, I was cautiously excited, so I broke cover and told my co-worker, Sava about the idea. He was initially suspicious about my Eval scores, which made sense because I accidentally sent him scores based on sentence chunking rather than paragraphs, where the NDCG difference is even more pronounced. It was much more reasonable when I corrected for that, but he did point out I was missing another baseline: what are the NDCG scores if you just used the summary and didn’t have any chunks at all? A great idea from our Principal Research Scientist and I collected the results:

Here’s the data formatted as markdown:

BOBBINS. I must confess I was somewhat crushed by this, and I still haven’t told him (until he reads this). I even went off and repeated the experiment with HuggingFace’s fineweb dataset…and…yes, as you can see, pretty much the same result.

Conclusion

All that work and I might as well have just used the summary. So is this a complete washout? Not entirely, and this is how even ‘failed’ research can still be useful. For one thing, instead of actually going to the trouble of doing all that chunking, it’s possible that for a variety of search applications, let’s just use not bother and use a summary? That way, we get good results and we don’t have to store all those chunk vectors in the database³

Also, there are some use cases where the averaged chunks could still be useful. In the currently common retrieval augmented generation pattern, chunks from various documents are sent to a large language model in order to answer a question. Even though today’s frontier models are powerful, they can still find themselves being confused by lots of different documents being sent to them in one go, so you want to send relevant information and not drown it in distracting text. If you’re trying to limit yourself to 5 or so docs to send, this technique gets you a much bigger chance of having the correct answer sent to the model…

There’s more that could be tested. We probably should have a better set of generated queries that can account for potential ‘bleed’ between queries and other documents by judging relevance of the query against other documents in the system (I have a version of this in progress as I type), as well as trying a bunch of other ideas. Should we try to append the summary in text form to the chunk text and then vectorize that new chunk? Just add title and maybe some important keywords? With those ideas, you do have to worry about total token counts again, but maybe if it’s kept concise and using a longer context embedding model, it would be easier to handle. Could we tailor the summary prompt to produce something more like an Anki card to see if that helps? Should we add a hyperparameter to multiply against the summary vector before it’s added and averaged against the chunk vector? Maybe we give up on the idea of the summary altogether and instead use a sliding window of previous and following chunks to try and keep context that way?⁴. So many different directions that could be ran into the ground. There’s still so much left on the table in the world of IR and embeddings. So, ‘failure’, yes, but as failures go, not a bad one.

March? I barely knew her!

Finally coming to the end of the flu. Still have the hacking cough and getting through a box of tissues a day, but it’s better than the start of the week.

Having said all that, still not a lot to talk about. I have finally done my taxes, although with the current state of the IRS, I’m not exactly convinced my refund will be coming to me any time soon. It’ll be nice once it does arrive though, as it’ll move me a big step closer to Weird Financial Goal.

I am still trying to work out Maeryn’s birthday cake. The basic contours are set: chocolate cake plus a raspberry component. But do I make it in the Berlin mold, which will be fancy, but I feel like it doesn’t have enough cake, as well as making a ganache layer much harder. Or I could use Kyoto and have a ganache filling and raspberries on top. It feels a more substantial cake size, but not overwhelming for toddler hands (I am fully aware that I’m over thinking this, and it could just be a box cake in a sheet pan and she’ll shove it into her face just the same, but left me have this!). Don’t worry, I still have two weeks left to figure it out…

First tech blog of the year has most of the first draft done. Unfortunately, it requires a vast amount of changes in the second draft, turning it from “look at this cool new technique” to more of “look at this idea that didn’t actually work out. Which is a little less fun to write up, and I need to do a few more experiments to convince myself one way or another, but I do feel it’s important to write up the things that fail as well as the things that succeed. So stay tuned for failure!