Comment by olliepro

This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.