But the thing that makes this special, is explicitly cutting things out that make LLM's capable of that. As gpm's sibling comment points out, if you would add that back in, you would just end up in the converging to the same place (with the same downsides).
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.