We ran a bunch (2,725) of Tessl verifier tests which are part of our test suite for our internal code base and switched the judge model to Jev and GPT Luna 6
The results:
- Jev is 13.6x faster
- Jev is 2.7x cheaper
- The models had a 85.9% agreement on verdicts
Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.
The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.
The results:
- Jev is 13.6x faster - Jev is 2.7x cheaper - The models had a 85.9% agreement on verdicts
Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.
The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.