I recently measured something similar on whether a quote supports the claim it's cited for. I tested on 240 examples from ALCE (labeled by human annotators).
Jev 1.13 flagged 87.5% of the citations that the annotators marked as not supported. The best regular LLM judge I tried was GLM-5.3-flash with reasoning which flagged 83.9% (without reasoning it was 75.6%).
The same also works with open models though. Give Qwen3.5-397B the three possible answers, let it output a single token, and read off the probability of each answer. No reasoning, one token per check. That flagged 89.3%.
There were only 56 unsupported cases, so these numbers are within each other's error bars. But my conclusion was that the speed and the confidence score come from reading out a single token, not from Jev itself.
Jev 1.13 flagged 87.5% of the citations that the annotators marked as not supported. The best regular LLM judge I tried was GLM-5.3-flash with reasoning which flagged 83.9% (without reasoning it was 75.6%).
The same also works with open models though. Give Qwen3.5-397B the three possible answers, let it output a single token, and read off the probability of each answer. No reasoning, one token per check. That flagged 89.3%.
There were only 56 unsupported cases, so these numbers are within each other's error bars. But my conclusion was that the speed and the confidence score come from reading out a single token, not from Jev itself.