You aren't wrong. In that mistakes can be made. But if I ask you about your left hand, is that ambiguous? By your definition here, that is an ambiguous question. But, that just feels wrong.
Edit: I'll note that it is amusing to compare to cars. It seems the etymology of the terms is literally "driver side".
Edit: I'll note that it is amusing to compare to cars. It seems the etymology of the terms is literally "driver side".