Comment by ianalloway1

How do you calibrate the error-finding model against human labels? My worry with judge models is that they share the agent's blind spots, since both were trained on similar data. Do you see a measurable gap between what the model flags and what a human reviewer flags on the same traces?