Good question, these are the two metrics we obsess over:
False accepts (bad response passed as good): <1% on benchmarks, ~2-3% in production pilots. This is the one that matters, we tune aggressively to keep it low. Every validator errs on the side of escalation.
False escalations (good response unnecessarily escalated): ~7-10% depending on domain. Costs you tokens, but doesn't hurt quality. The self-learning engine reduces this over time as it learns your traffic patterns.
The tradeoff is intentional: we'd rather waste some spend than serve bad answers. In practice, even with the conservative tuning, customers still see 30-60% cost reduction because the baseline.
False escalations (good response unnecessarily escalated): ~7-10% depending on domain. Costs you tokens, but doesn't hurt quality. The self-learning engine reduces this over time as it learns your traffic patterns.
The tradeoff is intentional: we'd rather waste some spend than serve bad answers. In practice, even with the conservative tuning, customers still see 30-60% cost reduction because the baseline.