I have worked on numerous full text search systems and I’d point out that the original Jeopardy playing Watson had the fairly unusual feature of consolidating multiple question answering methods with a probability score for the answers. This made it possible for Watson to decide if it was in its best interest to buzz the buzzer or not on a rational basis.
Outside of financial systems where people expect to put a Kelley better in front of a predictor I think this kind of thinking is pretty rare, but there are cases where a mediocre predictor that is calibrated can beat a better predictor which is not calibrated.
Outside of financial systems where people expect to put a Kelley better in front of a predictor I think this kind of thinking is pretty rare, but there are cases where a mediocre predictor that is calibrated can beat a better predictor which is not calibrated.