I have relatively poor hand eye coordination/balance and I've been trying to learn how to pilot a drone (without stabilization). Anecdotally, kids can pick it up within hours of flying a sim, and adults take in general 20-30 hours to get proficient. I'm double that and I'm still terrible.. flying a helictopter in the conditions you describe must be orders of magnitude more difficult.
Out of curiosity what do you do now?