zwaps

This is a fine-tuned model. The author even states that the model is competitive with Jev only if fine-tuned on the evaluation at hand.

Literally misses the point of Jev, which you don't need to fine-tune to get accuracy nor - and no other model has this - some sort of out of sample calibration

Gary Marcus has has a long career now of shifting goalposts

Such a crazy change from the times of Luminous, when they published a three pager with a claim that the model is similar good as „gpt 3“ (which??) with some graphs without y axis.

Bravo team!

No mention of calibration. Is it just another llm finetune?

There is museum in London called the British Museum which has zero artifacts on display from Britain and is entirely filled with stolen goods from other countries No one there finds this strange, bad, or anything that should be adressed

Brits are entirely deranged when it comes to their former empire.

You are holding it wrong!

I am sympathetic but this buries the lede, hard.

You are competitive with Jev only if you fine tune on the train dataset and calibrate per question.

As much as I dislike literally everything about typesafes behavior, they have an API model that works on any problem without fine tuning, and that is the key.

To be honest everyone who can finetune can likely finetune a BERT for a specific task and get similar results to yours. And that has been true for years

The key to Jevs success is that it works without fine tuning

Hey, don’t copy my daily affirmations

Thanks Claude for writing that

I literally struggle to read it

If the authors are reading: Why did you let an AI model author parts like "Design Principles"? It's not good writing and it is obvious.

But then Sol doesn’t really follow the plan or rather invents things it should add or cut on top of the plan. How does one deal with that?

Is anyone else having immense problems with this website due to Claude writing it?

Beyond the AI writing style and the pure amount of sentences, nothing seems to transmit the right information at the right time. You need to read the entire page twice to get what is going on.

For example, the first text you read is: —- Axtary checks the exact diff, message, query, or tool payload before a connector executes. Routine actions follow policy; higher-risk actions require approval of that exact payload. —-

Without having read the rest, nothing in this paragraph seems to tell me anything.

Only after I read the rest of the website do I somehow get what is even meant.

Sol is a complete mess for me.

It only works on end to end tasks in fresh codebases.

Otherwise it cannot follow instructions, changes and deletes unrelated features or does sloppy work to mark a task completed while leaving a compromised codebase.

I could not get Sol to finish a feature in a complex code base without several loops of fixing and reverting

Gpt 5.6 in codex is unusable for me.

It is no longer able to execute well defined changes. It does stuff that wasn’t asked for, deletes pieces of functionality unrelated to the task and introduces regressions everywhere.

I blame it on benchmaxxing. I fear coding models no longer work in a large complex codebase