A middle ground would be an LLM taking structured inputs ("mouse clicked at X,Y") and making "draw" calls at whatever level: pixels, SVG, HTML/CSS. While the LLM's output tokens could be mapped to just these calls, it could also just do a bunch of tool calls that write to these formats as well -- it's a matter of efficiency.
We have Jev to prove LLMs that output constrained decisions are useful. Why not make LLMs just output UI natively?
At a fast enough inference speed, you could just connect the network to it.
So an LLM would then just take the network, user input, and output a UI, with tool calls to deal with memory and other resources. In theory, you don't need an OS any more.
We have Jev to prove LLMs that output constrained decisions are useful. Why not make LLMs just output UI natively?
At a fast enough inference speed, you could just connect the network to it.
So an LLM would then just take the network, user input, and output a UI, with tool calls to deal with memory and other resources. In theory, you don't need an OS any more.