The problem is that deep net classifiers in general are not well statistically calibrated by default. So while the entropy is often high when they are "not sure", models can very often also be "confidently wrong". So using entropy of the logits as an indicator of confidence can easily be very misleading.
I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.
I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.