I think the idea behind agents is to have them a bit less constrained.
If you can force input/output with better prompts that might help.
Or if you can reduce the temperature of the underlying LLM generation, that could make it a bit more deterministic.
Or switch to using a chain.