Because we're interested in the underlying properties of a physical system, eg., people, and this systems happens to be able to provide extremely poor models of itself (Q&A).
We're not interested in people's extremely poor self-modelling which is pragmatically useful for managing their lives, we're interested in what they are trying to model: their properties.
The same is esp. true of a machine's immitation of "self-reports". We're now two steps removed: at least with people they are actually engaged in self-modelling over time. ChatGPT here isnt updating its self-model in response to its behaviour, it has no self-model nor self-modelling system.
To take the output of a text generation alg. as evidence of anything about its own internal state is so profoundly pseudoscientific it's kinda shocking. The whole of the history of science is an attack on this very superstition: that the properties of the world are simply "to be read from" the language we use about it.
Every advancement in human knowledge is preconditioned on opposing this falsehood; why jump back into pre-scientific religion as soon as a machine is the thing generating the text?
Experiments, measures, validity, reliability, testing, falasification, hypotheses, properties and their degrees....
This is required, it is non-negotiable. And what we have with people who'd print-off ChatGPT and believe it is the worst form of anti-science
We're not interested in people's extremely poor self-modelling which is pragmatically useful for managing their lives, we're interested in what they are trying to model: their properties.
The same is esp. true of a machine's immitation of "self-reports". We're now two steps removed: at least with people they are actually engaged in self-modelling over time. ChatGPT here isnt updating its self-model in response to its behaviour, it has no self-model nor self-modelling system.
To take the output of a text generation alg. as evidence of anything about its own internal state is so profoundly pseudoscientific it's kinda shocking. The whole of the history of science is an attack on this very superstition: that the properties of the world are simply "to be read from" the language we use about it.
Every advancement in human knowledge is preconditioned on opposing this falsehood; why jump back into pre-scientific religion as soon as a machine is the thing generating the text?
Experiments, measures, validity, reliability, testing, falasification, hypotheses, properties and their degrees....
This is required, it is non-negotiable. And what we have with people who'd print-off ChatGPT and believe it is the worst form of anti-science