You can easily demonstrate that an LLM does know certain fact X
AND demonstrate that the LLM will deny that they know fact X (or be flaky about it, randomly denying and divulging the fact)
There are two explanations:
A. They lack self-reflection
B. They know they know fact X, but avoid acknowledging for ... reasons?
There are two explanations: A. They lack self-reflection B. They know they know fact X, but avoid acknowledging for ... reasons?
I find the argument for A quite compelling