Post-training of LLMs often destroys calibration. The objective is no longer to predict the next token, so it doesn't matter as much whether the probability you assign to next-token "A" is correct, as long as the overall answer is good.
So it's not too surprising that the most heavily post-trained frontier models are actually pretty bad at outputting calibrated decisions.
So it's not too surprising that the most heavily post-trained frontier models are actually pretty bad at outputting calibrated decisions.