You're right but the trick LLM-backed tools have is that they can just write traditional code on the fly.
There are usability and performance problems right now which is why we have this interregnum. But as these systems evolve, the tools can offload "expensive" operations to traditional compute by writing the glue code on demand, and persist the code for future use (so regenerating it won't be required again later) when the user confirms it's doing what it's supposed to do. You can A/B test it in real time and ask the user to pick the "better" result.
Same is true for e.g. GUI elements, widgets, etc. Write it the first time, have your user verify it, persist it, retrieve it.
And maybe you have a couple of people in your org who work through the process of generating a bespoke app that processes data with a GUI optimized for whatever they're doing, but it's all mediated directly through your vendor. The users run through the process all in the web UI of whatever LLM vendor the org's data is silo'd with, the output is a mix of traditional computation and LLM contexts that anybody can maintain via asking the tool to change itself.
There are usability and performance problems right now which is why we have this interregnum. But as these systems evolve, the tools can offload "expensive" operations to traditional compute by writing the glue code on demand, and persist the code for future use (so regenerating it won't be required again later) when the user confirms it's doing what it's supposed to do. You can A/B test it in real time and ask the user to pick the "better" result.
Same is true for e.g. GUI elements, widgets, etc. Write it the first time, have your user verify it, persist it, retrieve it.
And maybe you have a couple of people in your org who work through the process of generating a bespoke app that processes data with a GUI optimized for whatever they're doing, but it's all mediated directly through your vendor. The users run through the process all in the web UI of whatever LLM vendor the org's data is silo'd with, the output is a mix of traditional computation and LLM contexts that anybody can maintain via asking the tool to change itself.