vessenes
Joined 17 years ago
Joined 17 years ago
Do they work with frontier labs? Probably. They should work with them at least a little so that we can get an independent read on the model quality. Do they inject new tests that specifically favor one particular lab? Clearly no, as different labs have moved up over time. Do they inject new tests that help to provide some separation on scoring? God, I hope so.
Nota bene, another change they announced was dramatically increasing the amount of held back test data; this would be a simple explanation for seeing open weights models, all of which are known to distill from frontier labs and benchmarks, drop in the rankings.
That’s an amazing amount of revenue and spend, though. Wild.
But then you have to compare to buying a GB200, sticking it in a closet and sharing it out in a workgroup and having a lower specced laptop. Decisions..
I’ve lost more than $1bn twice - failure is the best teacher! He says to himself..
[email protected]