Blocked Curtis

But This Was A Fantasy is now back up again, having spent much of the week reporting Service Unavailable. This came as something as a surprise to me, as the Docker container that contains the website hasn’t changed since May. Which meant the prospect of trying to debug Google’s Cloud Run substrate with no visibility! Hurrah!

It’s back now, so what was the problem? Honestly, I don’t really know, but I have a theory. Behold my debugging process!

• Poke at the site every day or two to see if it was a transient error that Google would fix
• Delve into the logs and see a permissions error trying to read /dev/urandom
• Run the container locally and confirm that it still worked there (yes!)
• Create a container with no images and try that
• Revert to an earlier container, and eventually all the old containers, discovering that they all now failed
• Try to remember how the base FastAPI container works, poke around in the directory structure and confirm I had permissions on /dev/urandom
• Mess around with the container’s running user
• Discover that the FastAPI container offers the ability to run a script at launch before the app (mainly meant for migrations). I stick an ls -l /dev/urandom in the script to see what I permissions I have in Google Cloud Run
• The website then starts fine
• Drink lots of alcohol

What I think happened is this: behind the scenes, Google has made some changes on how it handles random numbers in Docker containers. /dev/urandom should not block, but maybe it’s throwing an error because the system doesn’t have enough entropy during the container boot yet, and that’s manifesting as a permissions error. However, the ls -l, despite it not doing anything explicit to alter the system, gives it enough time to generate enough entropy for PyTorch to load fine.

Computers, eh?

Not much else I want to talk about this week, but stay tuned for a pastry book review that resulted from a hate-read of a quite expensive book (at least in printed version) this weekend.