One downside of training on TPUs is that by default they save models to GCE buckets. When buckets go down, there’s no way to save or load your models. It’s kind of funny: my model is training right now; I see it training right now; but it can’t save, so the progress will be wiped out if the TPU preempts or malfunctions.
Luckily they’ll fix the issue within 24h, which is when TPUs preempt. :)
Luckily they’ll fix the issue within 24h, which is when TPUs preempt. :)