Comment by wilson090

We separated out the runbook such that each step is a separate LLM in the agent. Between each step, there's sort of a "supervisor" that ensures that the step was completed correctly, and then routes to another step based on the results. So in reality, a single step failing requires two hallucinations. Hallucinations are also not a fixed percentage across all calls -- you can make them less likely by maintaining focused goals (this is why we made runbooks agentic rather than a single long conversation)