Forget about whether the ‘latent knowledge’ in the LLM is ‘correct’ or not, this kind of research is at least predicated on the idea that the LLM holds certain things as true, and consistently holds their opposite as false.
The point of this kind of work is to query the LLM and figure out what it ‘thinks is true’ - which is then presumably useful for deciding how much to rely on its output.
I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place.
Like, you just said that a language model ‘probably… holds representations of spurious facts and relations’.
But… isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something?
The point of this kind of work is to query the LLM and figure out what it ‘thinks is true’ - which is then presumably useful for deciding how much to rely on its output.
I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place.
Like, you just said that a language model ‘probably… holds representations of spurious facts and relations’.
But… isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something?