It would be great to see benchmarks for Jev that demonstrate the value of calibrated uncertainty.
For example, one could set a confidence threshold over which we trust the model decision, and otherwise reject. This provides a lever to trade-off accuracy and automation %.
Then we can ask questions like "What % of decisions can we automate to achieve 90% accuracy"?
For example, one could set a confidence threshold over which we trust the model decision, and otherwise reject. This provides a lever to trade-off accuracy and automation %.
Then we can ask questions like "What % of decisions can we automate to achieve 90% accuracy"?