Comment by arugulum

It is no coincidence that EleutherAI named their pretraining dataset "the Pile"