Comment by godelski

  > However, in my naïvety, I wonder whether vastly simpler algorithms could be used to end up with similar results.

Almost certainly. Distillation demonstrates this. The difficulty is training. It's harder to train a smaller network and harder to train with less data. But look at humans, they ingest far less data and certainly less diverse data. We are extremely computationally efficient. I guess you have to be when you run on meat