ActivePattern

Because a math proof is generally de-composable into smaller constituent arguments that can be independently evaluated for correctness. So you can audit a sample of these arguments for correctness without reviewing everything. You would then verify that the arguments being true implies the conclusion.

But the article clearly describes how this SCM is just instrumental in their goal to build and serve better models. It's not some unrelated product they hope to sell.

Post-training of LLMs often destroys calibration. The objective is no longer to predict the next token, so it doesn't matter as much whether the probability you assign to next-token "A" is correct, as long as the overall answer is good.

So it's not too surprising that the most heavily post-trained frontier models are actually pretty bad at outputting calibrated decisions.

The biggest upside to using a zero-shot classification model is that it's actually zero-shot, i.e., you don't need to construct task-specific training sets.

If it needs to be fine-tuned, then why not fine-tune smaller and cheaper models that are only as big as the task requires?

That's right. There's only so much optimization that you can make to a transformer-based model and any tricks that Jev is employing, any open-source LLM can also employ.

It's just marketing. More specifically, it's an answer for why their model can't answer questions that require reasoning.

"Reply with just the letter A, B, or C."

There, I fixed your problem.

It would be great to see benchmarks for Jev that demonstrate the value of calibrated uncertainty.

For example, one could set a confidence threshold over which we trust the model decision, and otherwise reject. This provides a lever to trade-off accuracy and automation %.

Then we can ask questions like "What % of decisions can we automate to achieve 90% accuracy"?

100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent.

If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.

Yeah, the README seems like AI slop to me. That always makes me lose interest in a project because it seems so low-effort.

How are you supporting the claim that GPT 5.6 is "far more token efficient" than Opus 5? Tokens equal, output is cheaper for Opus 5 ($25/1M) than GPT-5.6-Sol ($30/1M), and it seems to outperform slightly on agentic coding benchmarks.

I think you're being overly cynical here. First, I don't see any claim that is the world's best model for agentic coding. Second, it is absolutely the best model in terms of coding performance vs. dollar, and it's raw performance seems very close to the frontier.

It's interesting that it cares about formatting choices like bullet points.

On the one hand, it makes the system more brittle and arguably overfit, since it's not making a decision on the content of the writing itself. Most people would say that AI writing is still AI writing even if bullet points are removed, and vice versa.

On the other hand, we know that AI writing does has a very specific formatting signature (e.g. em-dashes, bullet points) so it seems unwise to completely ignore it.

You would be surprised how much of a difference the model makes for certain niche tasks.

For my use case, `gemini-3.1-flash-lite` is ~20% higher accuracy than the next best model of comparable cost (considering both proprietary and open-weight alternatives)

A lot of people in this thread think they're being clever by pointing out that a bookstore can't sell "banned" books. But it's common for bookstores and libraries to feature titles that have been banned in some jurisdictions. It's a small way to push back against censorship and promote freedom of information and critical thinking.