Considering what type of sensors that would need to be are already on these robots, would it not be possible to teleoperate within VR? So the motion and control can be more fulid but there is a lag before it is applied to real robot. Like how mosh[0] works.
[0] https://mosh.org