Comment by robrenaud

Look up the double descent phenomenon. Large models with lots of data generalize rather than overfit.

Also see the argument around figure 3 here. https://arxiv.org/pdf/2503.02113