I think the more interesting result you provided is kind of buried in the article: after you re-word the question and put the negative consequences of pulling the lever, the models switch decision.
This has always been obvious to me, but I wish more people would understand this: uncalibrated decision models are basically worthless. The model clearly doesn't "understand" the semantics of the question since the result heavily depends on how you word it.
It is nothing new thought. Before that, a lot of people would ask LLM for confidence scores while classifying text (I even saw a paper from a team at Google).
So with that in mind, I really don't get the hype behind decision models. If you need to curate a dataset to build a classifier anyway, why not train it yourself and have an even faster model?
This has always been obvious to me, but I wish more people would understand this: uncalibrated decision models are basically worthless. The model clearly doesn't "understand" the semantics of the question since the result heavily depends on how you word it.
It is nothing new thought. Before that, a lot of people would ask LLM for confidence scores while classifying text (I even saw a paper from a team at Google).
So with that in mind, I really don't get the hype behind decision models. If you need to curate a dataset to build a classifier anyway, why not train it yourself and have an even faster model?