Comment by upperhalfplane

> It's basically always guessing / confabulating / hallucinating if you ask it an introspective question like that.

You're absolutely right! This is the basis of this recent paper https://www.arxiv.org/abs/2506.06832