Comment

Comment by adi1

The breakdown with which model, per-task cost, and methodology is in the "Full results and methodology" link in the post, not omitted. Definitely check it out if you haven't.

On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.

parent adi1 2 months ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton