I'd guess that prompt injection is the biggest risk in this setup, dwarfing the risk of exploits against the inference engine. Personally, I run LLM agents only inside a Docker container that limits the LLM's access to sensitive information and the LLM's ability to take irreversible destructive actions.
See also: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/