I keep going back and forth on this.
Traces and logs tell me what the agent did: every tool call, retry, token, and step.
That's useful when something breaks. What I'm less sure about is whether that tells me the agent actually did the right thing.
A run can look clean and still miss what I wanted. Or it can "finish" without the outcome really being done. So finished and succeeded aren't the same thing.
I haven't made up my mind how to check for that, but I'm experimenting a lot with extracting intents from my agent and correlating among them.
Curious how you do it:
- Do you capture intent and outcome explicitly (e.g. giving a tool to the agent to "declare" it), or infer them after the fact from the traces?
- What kind of "questions" do you "ask" your traces?
Traces and logs tell me what the agent did: every tool call, retry, token, and step.
That's useful when something breaks. What I'm less sure about is whether that tells me the agent actually did the right thing.
A run can look clean and still miss what I wanted. Or it can "finish" without the outcome really being done. So finished and succeeded aren't the same thing.
I haven't made up my mind how to check for that, but I'm experimenting a lot with extracting intents from my agent and correlating among them.
Curious how you do it:
- Do you capture intent and outcome explicitly (e.g. giving a tool to the agent to "declare" it), or infer them after the fact from the traces?
- What kind of "questions" do you "ask" your traces?