We ran into an interesting side effect of agent performance that we needed to explain in our results to our executive team. In our case we were rolling out an AI support agent that could help resolve customer problems. At the start we had calculated the total time and human cost per support ticket going back a few years. After a few months of running the AI agent, we found that both our time and cost per ticket had increased - but then digging in, the issue was that our overall cost and time had decreased and it was just that the more complicated issues the agent wasn't able to answer were the ones now going to the humans, skewing our per ticket cost up.
So in this case it's not that it was a different failure, it was that a signal that seemed to indicate failure was actually a signal of success.
So in this case it's not that it was a different failure, it was that a signal that seemed to indicate failure was actually a signal of success.