I wonder if the entire thing would be better if the first step was for the LLM to try to understand the final goal of the user, rather than just the simple instruction.
For instance, the LLM may have an understanding that the ball is easier for it to pick-up than the elephant (which it seemed to struggle with, which also made it a strange choice for the demo).
How much of an LLM understanding the physical limitations of the robot would help the task be accomplished in a way that satisfies the goal, without following the exact directions.
For instance, the LLM may have an understanding that the ball is easier for it to pick-up than the elephant (which it seemed to struggle with, which also made it a strange choice for the demo).
How much of an LLM understanding the physical limitations of the robot would help the task be accomplished in a way that satisfies the goal, without following the exact directions.