Comment by dave_sullivan

Yes, but they didn't release the model after training and you can't take your weights with you if you finetune their model.

GPT Neo was trained at similar expense, and they released the weights. Use that.