Comment by foobarbecue

And the answer is: LLMs don't know ANY answers. They just know what answers tend to sound like. So it's a fundamental problem.