For the fine-motor commands: or, the model can write the code to generate them on the fly. It seems to work, in my very limited experiment.
As for memory: my approach is to give the robot a python repl and, basically, a file system - the LLM can write modules, poke at the robot via interactive python, etc.
Basically, the LLM becomes a robot programmer, writing code in real-time.
As for memory: my approach is to give the robot a python repl and, basically, a file system - the LLM can write modules, poke at the robot via interactive python, etc.
Basically, the LLM becomes a robot programmer, writing code in real-time.