For Opus, I don't really buy that xhigh is worse that medium. When scores are ~15%, it's unclear how much is just in the noise. The only things in favor of medium are the score and price: strict goes from (xhigh -> medium) 25 -> 28 and core goes 141 -> 145 out of 196. xhigh is better on strict, opus medium has _by far_ the most leftover code, callables, and not poor complexity of those callables.
If cost and score are the only things that matter, maybe medium is slightly better for a software factory. But I'd be curious if there were tasks in this benchmark that had twice as many checkpoints if medium's excess code would make it compare worse with xhigh.
If cost and score are the only things that matter, maybe medium is slightly better for a software factory. But I'd be curious if there were tasks in this benchmark that had twice as many checkpoints if medium's excess code would make it compare worse with xhigh.