Comment by hbrn

Yeah, but I wouldn't be surprised OpenAI's decision API is a post-trained Luna with confidence calibration.

Typesafe claims that Jev is calibrated, but there are plenty of examples where it completely fails (predicting die roll being the most obvious one).

Unfortunately calibration is hard to benchmark.