Comment by ole_tange

A good start is looking at the section 'Remote execution' in 'man parallel_tutorial'.

Spoiler: You can have GNU Parallel kick the machines if the fail, and rerun the job on another machine.