A technical person (such as yourself, presumably) can make an informed tradeoff, but a non-technical person i.e. the general public as OP described just sees it as a black box that makes purchases and appointments, and then gets surprised when their bank details end up in Slack[0], hacks a local gym[1] or the agent sends a stranger their address[2].
[0] https://www.businessinsider.com/personal-ai-agent-grok-bot-p...
[1] https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
[2] https://www.pcmag.com/news/muse-ai-shared-address-error-trac...