I am puzzled why they have "asked the model" about the confidence and have not used the logprobs of the output tokens to estimate the confidence in responses.
In my use case and tests, model itself is not capable of giving a reliable confidence value where logprobs almost always provide a better view on calibration.
In my use case and tests, model itself is not capable of giving a reliable confidence value where logprobs almost always provide a better view on calibration.