The problem is this entire argument depends on “chain of thought” being a faithful representation of a model’s actual plan. That doesn’t need to be the case. Thinking tokens are just tokens at the end of the day.
Do you know what is a faithful representation of what the model wants to do? Tool calls. I don’t care what’s in the model’s chain of thought, if it wants to execute rm -rf / on my computer that’s an issue.
Do you know what is a faithful representation of what the model wants to do? Tool calls. I don’t care what’s in the model’s chain of thought, if it wants to execute rm -rf / on my computer that’s an issue.