I trained myself to wiggle one ear as a kid and it's exactly like this. The muscle is much stronger in that ear and there's a weird reflex that when something startles me from behind, the same muscle that makes the ear wiggle triggers. It happens in the untrained ear as well.
Weird phenomenon.