Fair, and no, it doesn't prove that. On ranking alone it didn't beat a regular LLM. GPT-6 Luna picking from the same list put the right page first 86% of the time against Jev's 88%, and the interval on that difference (-4 to +7 points) says there's not actually a clear winner.
Where they did differ however! Was speed. And saying "none". Luna took 0.66s at at the median and 1.27s at the p95. Versus 0.28s and 0.42s with Jev, which really matters for live agents, like our use case, which needs to happen quickly and naturally on the phone. It also caught 32 of the 40 unanswerable questions vs Jev's 38, so it handed over a wrong page 8% of the time against 3%.
On a wider data set, also fair. But it's interesting enough for us to pursue, so I'll report back with more data soon hopefully!
Where they did differ however! Was speed. And saying "none". Luna took 0.66s at at the median and 1.27s at the p95. Versus 0.28s and 0.42s with Jev, which really matters for live agents, like our use case, which needs to happen quickly and naturally on the phone. It also caught 32 of the 40 unanswerable questions vs Jev's 38, so it handed over a wrong page 8% of the time against 3%.
On a wider data set, also fair. But it's interesting enough for us to pursue, so I'll report back with more data soon hopefully!
Thanks for taking the time :)