Gating completion on the observation write is smart — you're turning the model's task-completion drive against itself. Have you run into the problem where forced observations degrade to "completed task successfully, no issues noted" though? Technically passes the schema, zero actual information.
That's what killed the approach for me. I spent weeks tuning schemas and rejection criteria and the models just got better at producing plausible-sounding observations that said nothing. Passive extraction ended up more reliable — watch the AST diffs, infer what the agent learned from what it actually changed, skip the self-report entirely.
Curious what your checker validates against. If it's structural completeness of the fields you'll hit the gaming problem fast. If it's semantic quality... how? That's basically asking another model to judge whether an observation is useful, which is its own rabbit hole.
That's what killed the approach for me. I spent weeks tuning schemas and rejection criteria and the models just got better at producing plausible-sounding observations that said nothing. Passive extraction ended up more reliable — watch the AST diffs, infer what the agent learned from what it actually changed, skip the self-report entirely.
Curious what your checker validates against. If it's structural completeness of the fields you'll hit the gaming problem fast. If it's semantic quality... how? That's basically asking another model to judge whether an observation is useful, which is its own rabbit hole.