Comment by bigmadshoe

Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.