carlsverre
Joined 15 years ago
Karma
470
Submissions
158
Member since
May 12
Recent submissions
Comment · 2 months ago
Oh man, Beyond All Reason would be a killer demo. We are actually testing another RTS as we speak, but I won't spoil it for you :) Look forward to a really epic post coming soon.
Comment · 2 months ago
We've started testing git and DuckDB! Haven't played much with Dolt, though. We have some initial findings on Git that we are validating and will hopefully be making public soon!
Comment · 2 months ago
Thank you! I appreciate the note of support! I'd love to write more blog posts in the future.
Comment · 2 months ago
We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year (https://etcd.io/blog/2025/autonomus_testing_with_antithesis/) and are working with various OSS groups to put more systems under test in a responsible way.
I'm not gonna lie, though, you're hitting a sore point. I'm absolutely kicking myself for not putting SQLite under test when I joined. The first few projects I put under test after joining were all projects that had a few properties: 1. The project had a path to a commercial deal 2. The project likely had interesting bugs to find
TBH, I didn't expect that one of the most widely used and well-tested pieces of software in existence would have a bug our fuzzer could find in 15 mins. (To be clear, this is not a hit on SQLite, just that our fuzzer is stupid good at finding race conditions. SQLite remains one of my favorite pieces of software ever.)
With this in mind, are there other similar projects that I'm not thinking of that I should put under test? Before you ask, I'm already scaling up my SQLite harness to cover more of its state space and run against master. Maybe I'll find something novel in SQLite after all! Never too late to check.
Comment · 2 months ago
I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach:
https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521
Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.
Comment · 2 months ago
Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.
Comment · 2 months ago
This is a solid idea. However, something that is maybe similar to this idea is this: https://antithesis.com/blog/2026/finding-bugs-in-raft-implem...
Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.
Comment · 2 months ago
Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis.
The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
Comment · 2 months ago
Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing other WAL checkpoint-style systems: by putting them under stress. The workload itself does not know about the bug.
My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test.
Comment · 2 months ago
Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug
I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.
If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
Comment · 2 months ago
Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.
Comment · 4 months ago
Mostly posting for entertainment. The issues on the project are fun:
https://github.com/TiruArt/Pedigree-Polytopes-Lean4/issues/1
Building SQLSync and Graft. https://sqlsync.dev https://github.com/orbitinghail/graft
[ my public key: https://keybase.io/carlsverre; my proof: https://keybase.io/carlsverre/sigs/5eMnLTR-fsM9h4DZqnQ2ZBkWfNxYyzKGI7265PNjWzE ]