#505: Introducing diskcache-oxide
39 min
Topics covered in this episode: tinyjobs: a SQLite job queue goes free-threaded funkycache: cache Python functions in SQLite Ghostman - a keyboard-driven terminal API client diskcache-oxide: A shared, durable disk cache Extras Joke Watch on YouTube About the show Sponsored by LongLink: pythonbytes.fm/longlink Connect with the hosts Michael: Mastodon / BlueSky / X / LinkedIn Calvin: Mastodon / BlueSky / X / LinkedIn Show: Mastodon / BlueSky / X Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesday at 7am PT. Older video versions available there too. Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Michael #1: tinyjobs: a SQLite job queue goes free-threaded tinyjobs is a Python job queue and cron scheduler that starts with a SQLite file and no dependencies. Switch the URL to Postgres or Redis without rewriting tasks. It runs async tasks natively, embeds workers in FastAPI, and plugs into Django 6's tasks framework. The creator's free-threading audit found four races hidden by the GIL. Background jobs without deploying a broker, plus a concrete case study in what removing the GIL exposes. The creator reports races in the task registry, codec cache, cross-thread flag and thread pool. hiredis can re-enable the GIL; the worker warns when it detects that. On 3.14t, --loops N runs event loops in separate threads. The author's laptop benchmark hit 42,000 async jobs/s with eight loops and two Redis instances, close to eight processes. SQLite rejects loops above one. Clock-skew lesson: two hosts 20 minutes apart duplicated 492 of 3,000 jobs on 0.7.0. Using Redis server time fixed that test, not the general exactly-once problem. Author's SQLite measurements: about 572 jobs/s end-to-end; a 20,000-job burst drained in four seconds. Synthetic workloads and different harnesses, not interchangeable throughput...