> I really am still failing to understand whats the big deal about our brains emitting language and thought and a LLM doing so.
Our brains "output" language as a byproduct (or rather shadow?) of conscious experience. An LLM is a machine built to project similar shadows, but that doesn't necessarily imply there's an entire world like ours shrunk inside that box.
Both brains and LLMs are physical, and both output language, but one is an appliance built to synthesize language and the other is a living being that merely uses it as a tool. LLMs are made out of pure language in a way people are simple not. Our minds undergo physical changes with every stimulus we process, we conceptualize ourselves separately from our environment without needing to be taught an ego, and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
The biggest issue is you assume the LLM has any "I" from pre-training. Take a pure completion model like GPT-3 and you'll see it's most appropriate to describe as a simulator, not an entity in itself. The model itself doesn't have any notion of an "I", it will gladly write dialog as any character, from Skynet to the user themselves, if you don't apply external rules to constrain it, first coming in the form of stop tokens and prompts, and having later evolved into RLHF, bashing a "personality" into it by mangling its existing probability distributions (which is probably why all the most "aligned" models write in the most unnatural language, worse than those early non-chat LLMs). Any persistent self an LLM demonstrates was explicitly imposed on it by humans, which makes them obviously categorically different from animals or disabled people.
Our brains "output" language as a byproduct (or rather shadow?) of conscious experience. An LLM is a machine built to project similar shadows, but that doesn't necessarily imply there's an entire world like ours shrunk inside that box.
Both brains and LLMs are physical, and both output language, but one is an appliance built to synthesize language and the other is a living being that merely uses it as a tool. LLMs are made out of pure language in a way people are simple not. Our minds undergo physical changes with every stimulus we process, we conceptualize ourselves separately from our environment without needing to be taught an ego, and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
The biggest issue is you assume the LLM has any "I" from pre-training. Take a pure completion model like GPT-3 and you'll see it's most appropriate to describe as a simulator, not an entity in itself. The model itself doesn't have any notion of an "I", it will gladly write dialog as any character, from Skynet to the user themselves, if you don't apply external rules to constrain it, first coming in the form of stop tokens and prompts, and having later evolved into RLHF, bashing a "personality" into it by mangling its existing probability distributions (which is probably why all the most "aligned" models write in the most unnatural language, worse than those early non-chat LLMs). Any persistent self an LLM demonstrates was explicitly imposed on it by humans, which makes them obviously categorically different from animals or disabled people.