The author (and Chomsky) fail to understand that LLMs (as well as human brains) are not just autoregressive models, but
nonlinear autoregressive models. Put a slightly different way, you can describe LLMs as autoregressive, but only by taking liberties with the classical definition of 'autoregressive.'
The human mind is not, like ChatGPT and its ilk, a lumbering statistical engine for pattern matching, gorging on hundreds of terabytes of data and extrapolating the most likely conversational response or most probable answer to a scientific question. On the contrary, the human mind is a surprisingly efficient and even elegant system that operates with small amounts of information; it seeks not to infer brute correlations among data points but to create explanations. – Noam Chomsky
It's as if Chomsky has either never heard of transformers, or doesn't understand what they do.
Before speaking a sentence, we have a general idea of what we’re going to say; we don’t really choose what to say next based on the last word. That kind of planning isn’t something that can be represented sequentially.
It's as if the author (and Chomsky) has never seen a CoT model in action.
The human mind is not, like ChatGPT and its ilk, a lumbering statistical engine for pattern matching, gorging on hundreds of terabytes of data and extrapolating the most likely conversational response or most probable answer to a scientific question. On the contrary, the human mind is a surprisingly efficient and even elegant system that operates with small amounts of information; it seeks not to infer brute correlations among data points but to create explanations. – Noam Chomsky
It's as if Chomsky has either never heard of transformers, or doesn't understand what they do.
Before speaking a sentence, we have a general idea of what we’re going to say; we don’t really choose what to say next based on the last word. That kind of planning isn’t something that can be represented sequentially.
It's as if the author (and Chomsky) has never seen a CoT model in action.