Comment by tianlins

The high dimensionality is merely for the convenience of optimization, so that we get smoother manifold on which a good approximation solution can be found. In that regard, it is interesting to think about of space of models that we humans can interpret, and the possibility of "distill" deep neural nets into models in that space.