Comment by tsvoboda

Would love to hear how you're handling recovery for long-running training jobs today, as well as what failure modes are most common/annoying for you.