mjb

And yet, the specification of almost every program in existence was communicated in natural language. At least between the customer and developer.

This argument makes the implicit assumption that humans are uniquely able to convert between this "customer wants a program that solves their problem" level and compileable code. That was a reasonable thing to believe in 1978, but is not a reasonable thing to believe in 2026.

Oh, huh, that's an error in the image I didn't notice! Those comparisons are to a model called 'decider-2B', which isn't ours.

These benchmarks cover strands-decider v19. v21 is better calibrated, and we've got some new ones coming this week that move us further towards the bottom right.

No particularly principled reason, no. It's one of the (many) design variants we haven't had time to experiment with yet.

One of the authors here.

I appreciate what the JevBench folks are doing, but it's true the JevBench (like all benchmarks) isn't representative of every real-world workloads. I particularly don't love how JevBench handles confidence, and rewards being highly confident in certain cases. Benchmarking is hard, and JevBench is probably more representation of its target workloads than TPC-C is :)

As for actual intelligence, there's only so much you can expect from 2B without reasoning. One of the design challenges in training this model was to avoid forgetting too much, and the KL-to-frozen-base step partially exists for that purpose. Even then, Qwen3.5 2B base has fairly limited single-pass reasoning ability (which shows up for us in the performance on the JevBench hard set).

It's very doable, but we haven't done it yet (although some great community folks did an ONNX version of v21).

I haven't looked in depth, but it should be doable without modifications to llama.cpp. You can't just convert the LoRA, through - there's a whole pointer head and some custom layers that need to be correctly handled (in fact, the LoRA mostly exists to get the base model to behave the way the pointer head needs).

All are viable options.

(1) would take the most data (probably, depends on your domain) and probably generalize worst, but would be closest to compute-optimal in the end.

(2) doesn't take much data, you can tweak calibration to your needs, and might only take a few minutes on a beefy GPU.

(3) is the way to go if you need a lot of general knowledge, any amount of reasoning, and don't need good calibration. Likely the least inference efficient of the three options for a given accuracy and calibration.

The hard part of (3) is keeping the answers to the questions independent. If you dump them all together with the state into the prompt, the answers to the second question will depend on the first (and the answer to the first if you it step-by-step). You can do it by tweaking the inference process with the right masking or use of batching, or by fiddling with cache control using a provider's API.

One of the authors here.

Yes, at this size (2B) you can get better performance and calibration fine tuning on your questions. You can also improve calibration on your problem type just be recalibrating without fine-tuning.

As to what's doable, that depends on what you expect. It's a single pass through a 2B model, no reasoning, so it's never going to be particularly 'intelligent'. On domain problems, though, you can get great in-distribution performance (and possibly better in-distribution performance than you'd get using the same volume of data to train a specialized classifier).

Everything you need to fine-tune, or even retrain from scratch, is in the github repo.

Inside multiple AWS products (including DynamoDB, Kinesis, and Aurora DSQL) is a system called Journal that moves a ton of data. It uses a variant of chain replication.

EBS is also a chain replication variant at heart, and moves even more data.