If you found this interesting, you should read section 5 of this article to learn about how LLMs
can be trained so that they're good at confidence scoring:
https://magazine.sebastianraschka.com/i/217935364/51-on-cali...OP says that LLMs were never scored against correctness. But that's an assumption about the optimization process.
This is sort of right:
- pretraining scores on next token, and says nothing about confidence
- RLHF and RLVR scores are binary (1=preferred/correct, 0=worse/incorrect) and the feedback doesn't do anything to push the weights in a direction that improves calibration/confidence
BUT you can ask the model to output its confidence, and then train it to be more correct. RLCR (mentioned in the article I linked above) adds a Brier penalty to the reward, so the model's performance isn't just measured on % correctness. (Presumably TypeSafeAI's RLCD is similar to RLCR, but they haven't published anything about it.)
This eliminates the "always say 30%" loophole OP mentioned. The Brier penalty, (q-c)^2, punishes a forecaster who ignores instance-level information. For example if we have a model that's right 60% of the time, it could use one of these strategies to give its confidence:
Strategy A, always say 0.6:
When right (60% of the time): (0.6-1)^2 = 0.16
When wrong (40% of the time): (0.6-0)^2 = 0.36
Expected penalty: 0.6×0.16 + 0.4×0.36 = 0.24
Strategy B: the model can tell its answers apart. Half are 'easy' (right 90% of the time) and half are 'hard' (right 30% of the time), which still averages to 60%. It says 0.9 on easy ones and 0.3 on hard ones:
Easy: 0.9×(0.1)^2 + 0.1×(0.9)^2 = 0.09
Hard: 0.3×(0.7)^2 + 0.7×(0.3)^2 = 0.21
Expected penalty: (0.09+0.21)/2 = 0.15
Both strategies are perfectly calibrated (60% is both the average confidence and thje average % of time it's right), but B gets a lower penalty. Of course these are not actual strategies used by LLMs. But I'm just trying to say why training can push the model toward optimizing
instance-specific confidence.
OP says that LLMs were never scored against correctness. But that's an assumption about the optimization process.
This is sort of right:
- pretraining scores on next token, and says nothing about confidence
- RLHF and RLVR scores are binary (1=preferred/correct, 0=worse/incorrect) and the feedback doesn't do anything to push the weights in a direction that improves calibration/confidence
BUT you can ask the model to output its confidence, and then train it to be more correct. RLCR (mentioned in the article I linked above) adds a Brier penalty to the reward, so the model's performance isn't just measured on % correctness. (Presumably TypeSafeAI's RLCD is similar to RLCR, but they haven't published anything about it.)
This eliminates the "always say 30%" loophole OP mentioned. The Brier penalty, (q-c)^2, punishes a forecaster who ignores instance-level information. For example if we have a model that's right 60% of the time, it could use one of these strategies to give its confidence:
Both strategies are perfectly calibrated (60% is both the average confidence and thje average % of time it's right), but B gets a lower penalty. Of course these are not actual strategies used by LLMs. But I'm just trying to say why training can push the model toward optimizing instance-specific confidence.