Comment

Comment by jbellis

It's not that hard to create your own uncontaminated eval for the problems you care about, and compare.

Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking

parent jbellis 3 days ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton