Comment by arcanemachiner

Devil's advocate: Their scoring system is itself a meta-benchmark.

What happens when you release a new benchmark? Models magically start scoring better against it.

How do you overcome this? Release a new benchmark and start the process over again!

All other things being equal, they should retarget their prices if models start gaming the process to top their index.

Has this happened? Maybe. Is their new scoring system biased towards the American frontier models? Quite possibly.

I have my suspicions, but they have led me astray before. I will continue to be cautiously cynical, though.