Comment by solidasparagus

You look at the inputs to the model at the trace level to root cause context construction vs model deficiency. You collect examples of the failure cases, build eval sets, and run ablations. That's black box debugging, but systematic debugging nonetheless.