Comment by vladf

This was done last year: https://ai.googleblog.com/2022/02/can-robots-follow-instruct...