My experience is that sota LLMs are pretty bad at confidence and priorities and things.
I write harnesses that do decision steps and regularly use competing models - from the major vendors and open source models.
And I have learned that if I can isolate a problem and put it into a straightforward prompt then Gemini 2.5 flash - the most basic and cheap commercial model - is actually very reliable. The newer models take longer, cost more and are really bad at saying they don’t know or nothing. You can ask them for a confidence score but they just hallucinate it!
I would like to try out decision models like Jev. The way the decision models approach the problems I work with seems a much more promising fit.
I write harnesses that do decision steps and regularly use competing models - from the major vendors and open source models.
And I have learned that if I can isolate a problem and put it into a straightforward prompt then Gemini 2.5 flash - the most basic and cheap commercial model - is actually very reliable. The newer models take longer, cost more and are really bad at saying they don’t know or nothing. You can ask them for a confidence score but they just hallucinate it!
I would like to try out decision models like Jev. The way the decision models approach the problems I work with seems a much more promising fit.