Comment by chengyongru

Models tend to default to their training data even when they lack sufficient context, they've never been trained to recognize their own uncertainty, so they hallucinate confidently instead.