While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
https://x.com/PawelHuryn/status/2095982259761475945
https://bughunt.productcompass.pm/?preset=all
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.