solohunter001

Updated the Readme!! Some new benchmarks available, Ty

Thanks for feedback, I will try doing this going forward

I was working with a large entries on Batch inference, and yes LLMs are quite expensive to process Billions of contracts at latency, so I in my orignial paper was on the expensive thinking once, run the cheap deterministic result forever.

Hi HN. STROT compiles a decision task — "route these tickets", "score this feedback", "classify this text" — into a standalone pipeline. A large LLM is used once, at compile time, to synthesize the labels, the input projection, and the handlers. At runtime a small local model (144M, CPU) makes every decision: ~60ms/record, $0 marginal, zero LLM calls, bounded typed-JSON output.

The goal was to move the expensive, nondeterministic part (an LLM) entirely to build time and keep runtime frozen, cheap, local, and auditable — the way a compiler moves work off the hot path.

What I think is actually interesting, beyond "small model does inference":

- The compiler measures instead of guessing. It scores candidate labelings on a teacher-labeled sample and keeps the winner; on the identical model that's +18pp on a routing benchmark (92.5% vs 74.8%) and +14pp on text classification (86.7% vs 72.7%) over naive integration. - It fits a conformal-calibration threshold at compile time, so low-confidence records abstain with a marginal error guarantee rather than guessing. - A golden-record gate rejects any compile that fails known-answer examples — synthesis is stochastic, and I show that variance in the benchmarks.

What it doesn't do: it can't make a small model understand things it fundamentally can't. On tweet hate-speech it hits the model's ceiling; the fix there is a task-trained model in the swappable slot, not the compiler. All of it — including the losses — is in the repo's benchmarks with reproducible scripts.

There's a browser demo (a 144M model ( which is changeable ) playing Pac-Man, one decision per frame) and a quickstart that runs in a few lines. MIT licensed, pure-stdlib core.

Happy to go deep on the calibration math, the compile/cache design, or where this does and doesn't make sense.