I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.
I must not have the right idea of what is happening here.
I must not have the right idea of what is happening here.