Anybody that doesn't acknowledge this as a base truth of these systems should not be listened to. It's not intelligence, it's statistics.
The AI doesn't reason in any real way. It's calculating the probability of the next word appearing in the training set conditioned on the context that came before, and in cases where there are multiple likely candidates it's picking one at random.
To the extent you want to claim intelligence from these systems, it's actually present in the training data. The intelligence is not emergent, it's encoded by humans in the training data. The weaker that signal is to the noise of random internet garbage, the more likely the AI will be to pick a random choice that's not True.
The AI doesn't reason in any real way. It's calculating the probability of the next word appearing in the training set conditioned on the context that came before, and in cases where there are multiple likely candidates it's picking one at random.
To the extent you want to claim intelligence from these systems, it's actually present in the training data. The intelligence is not emergent, it's encoded by humans in the training data. The weaker that signal is to the noise of random internet garbage, the more likely the AI will be to pick a random choice that's not True.