I remember when Opus 5.5 came out, we had a discussion about whether it was nerfed a week later [1]. It's not possible to do so with Gemini 4 models since none has been made available the day it is announced.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.