LLMs have shown a way to absorb large amounts of textual training data and be useful. For sensory processing you need a different architecture to absorb spatio-temporal streams. Bolting vision onto language models is stretching the tech a bit too thin.
The good news is the data is plentiful — it is all around (the robot). The real world — the robot’s umwelt sensed through its sensors and acted on by its actuators — will provide heaps of data.
Interestingly my colleagues propose that embodied learning will lead to grounding, alignment, and AI safety.
(Worked for 4 years at a robotics + machine learning startup, Director of R&D.)
The good news is the data is plentiful — it is all around (the robot). The real world — the robot’s umwelt sensed through its sensors and acted on by its actuators — will provide heaps of data.
Interestingly my colleagues propose that embodied learning will lead to grounding, alignment, and AI safety.
(Worked for 4 years at a robotics + machine learning startup, Director of R&D.)