Likewise, I switched over about 20 frustratingly buggy llm judgement calls that were using Luna (should the model use X tool, should it trigger deep research, what topic is the question etc.). I had spent days tuning the prompts to get somewhat decent results - meaning correct maybe 70% of the time.
I told an agent to test switching those all over to Jev. Came back a couple of hours later, thinking of this as an experiment, no major expectations, to find that they were all giving 100% accurate results in 800ms median time.
So it got adopted in a couple of hours work, it's live now, working very well, and I haven't had to think about it for the past week. That's the definition of a successful product.
EDIT: I should note two things
- there was actually one non English case that didn't have such high accuracy, now that I think about it.
- I assume a big part of why I got such good results was that I'd already spent ages tuning the prompts for Luna
I told an agent to test switching those all over to Jev. Came back a couple of hours later, thinking of this as an experiment, no major expectations, to find that they were all giving 100% accurate results in 800ms median time.
So it got adopted in a couple of hours work, it's live now, working very well, and I haven't had to think about it for the past week. That's the definition of a successful product.
EDIT: I should note two things
- there was actually one non English case that didn't have such high accuracy, now that I think about it.
- I assume a big part of why I got such good results was that I'd already spent ages tuning the prompts for Luna