kostaj

I'm Kosta, co-author of the research and founder of Lenz. Our goal is to assess to what extent the frontier LLMs are interchangeable as verifiers of factual claims. In this revision v1.1 of the research: improved methodology, latest frontier models (incl. Fable and Sol), added confidence analysis, public code.

Key findings (on a five-point True-False scale): on 63% of the claims at least one model dissents from the panel majority (or no majority at all); on 23%, the differences are significant, while the models are highly confident almost everywhere.

Some models struggle combining JSON schema and web search capabilities.

Good point. Will publish in the next version also the results with a prompt that allows the models to "think out loud" before providing the final verdict.

Awesome. We do plan to human-label the 1,000 claims and then compare Lenz' performance vs the 5 models. We've done some limited internal research with 150 claims, but more are needed for statistical significance.

Agree that some of the claims are forward-looking. The messiness of the real-world and real-user fact checks. No ground-truth verdicts are provided or used in the study though. It only measures the level of agreement between the selected models, not which one is right on which claim. I.e. none of the claims is actually labelled.

Good idea about publishing intra-model variance data! Will include in the next version. Even if we put aside the two middle buckets (Mostly True and Misleading), that are somewhat subject to interpretation and hedging: On 21% of the claims still at least two models provide polar-opposite verdicts (one model saying True, and another saying False)

Good point. Processing the substance of the answer might be too labor-consuming (1,000 claims x 5 models), but "thinking out loud" might improve the quality of the answers indeed. And we can still force/ask them to respond with a clear verdict at the end of their reasoning, as per the chosen rubric.

This is in line with my observations and tests as well. Also supported by the distribution of the verdicts across the 4-buckets -- Gemini uses the middle buckets (Mostly True and Misleading) much less often - 6% combined for Gemini w/o search. And Opus uses them the most - 45% combined. Looks like Gemini is calibrated to be confident and Opus to be careful.