senoff

So we did the obvious thing: moved CI to a single self-hosted box. Fixed monthly cost instead of a per-minute meter, and the builds actually got faster because the box keeps a warm Docker cache and doesn't cold-start every job.

It sounds like a five-minute job — spin up a VM, run config.sh, done. It mostly is. But three things bit us that nobody warns you about, and they're the reason we bothered to package this up:

1. OOM. A memory-constrained box running a couple of heavy jobs at once just gets its builds OOM-killed, silently, and you spend an afternoon blaming flaky tests. Fix: give the box swap — specifically one swapfile per runner — and set vm.swappiness=10 so it only leans on swap under real pressure. Swap isn't the hot path; it's the airbag.

2. "No space left on device" — but the disk looks half empty. The culprit was /tmp silt: dead job-workspace directories that never get cleaned up, slowly filling the disk until a big build tips over. Fix: give each runner its own /scratch (so they don't fight over one /tmp), and add a systemd-tmpfiles rule that ages /tmp + scratch every 6h. Set-and-forget.

3. The test database. Our suite needs Postgres. One shared instance every job writes into means parallel runs clobber each other's data and burn through connections; the other extreme, a fresh Postgres container per job, is heavier and slower than it needs to be. What worked: one small native Postgres on the box (128MB shared_buffers — deliberately tiny), shared by all runners, with each run isolated in its own schema — create it on the way in, point search_path at it via DATABASE_URL, drop it in an always-run cleanup step. Small-and-shared beats N heavy instances on a constrained box, and per-run schemas make concurrent runs safe without paying for a container spin-up on every job.

The operating rule we landed on for an 8 GB box: provision 3 runners, but cap concurrency at 2. The third is a warm spare and gives you headroom; running all three heavy jobs at once is how you meet the OOM killer again. Bigger box, bump both numbers — it's just policy.

We wrote it up as a few idempotent scripts + a README so it's reproducible instead of tribal knowledge: