> Unless otherwise specified, we evaluated o1 on the maximal test-time compute setting.
Maximal test time is the maximum amount of time spent doing the “Chain of Thought” “reasoning”. So that’s what these results are based on.
The caveat is that in the graphs they show that for each increase in test-time performance, the (wall) time / compute goes up exponentially.
So there is a potentially interesting play here. They can honestly boast these amazing results (it’s the same model after all) yet the actual product may have a lower order of magnitude of “test-time” and not be as good.
Maximal test time is the maximum amount of time spent doing the “Chain of Thought” “reasoning”. So that’s what these results are based on.
The caveat is that in the graphs they show that for each increase in test-time performance, the (wall) time / compute goes up exponentially.
So there is a potentially interesting play here. They can honestly boast these amazing results (it’s the same model after all) yet the actual product may have a lower order of magnitude of “test-time” and not be as good.