I'm a robotics noob currently building a prototype of a snow sweeping robot (www.frost-e.com) using my own design and I'm encouraged to see similar work coming from real places. Namely:
- Zero shot control using a visual harness and action primitives. I've seen this from a few other places as well (https://tml.stanford.edu/homebody/) and it's essentially having a generalist LLM command a robot via a set of predefined actions. Many robots and many tasks can probably be done with this approach, since not everything needs super intricate 6DoF arms or dexterity
- simulation bring-ups: I have a pipeline that takes in satellite imagery and reconstructs the lots and houses in a 3D environment and snows on it so that my virtual robot can learn to clean. With these latest models, you can probably create a higher fidelity digital twin more directly.
- self-improvement: These models (Argon, Astra, etc) can't operate the robot itself, but would be able to create data to distill into smaller models you can actually utilize.
- Zero shot control using a visual harness and action primitives. I've seen this from a few other places as well (https://tml.stanford.edu/homebody/) and it's essentially having a generalist LLM command a robot via a set of predefined actions. Many robots and many tasks can probably be done with this approach, since not everything needs super intricate 6DoF arms or dexterity
- simulation bring-ups: I have a pipeline that takes in satellite imagery and reconstructs the lots and houses in a 3D environment and snows on it so that my virtual robot can learn to clean. With these latest models, you can probably create a higher fidelity digital twin more directly.
- self-improvement: These models (Argon, Astra, etc) can't operate the robot itself, but would be able to create data to distill into smaller models you can actually utilize.