The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now.
The manifold hypothesis is that natural data forms lower-dimensional
manifolds in its embedding space. There are both theoretical and
experimental reasons to believe this to be true. If you believe this, then
the task of a classification algorithm is fundamentally to separate a bunch
of tangled manifolds.
My interpretation/rephrasing: if you want to build a neural network that distinguishes cat and dog pictures, in the worst case that would seem to require a huge network with many nodes/layers (say, the number being a function of the size of the image) rather than the number that seems to work reasonably well in practice (six or some other rather low constant number observed in reality). So the number of dimensions over which the “images” are potentially spread is huge, but it’d seem that in the real world one can rearrange the dog and cat images in a “shape” that then allows for relatively easy disentanglement by the neural network; and these shapes can probably be realized in much lower dimensions (in the example, six).
This could explain (for some definition of explain) the observed predictive power of relatively small neural networks.
https://news.ycombinator.com/item?id=7557964 https://news.ycombinator.com/item?id=9814114
But not a lot of discussion over there.
The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now.
My interpretation/rephrasing: if you want to build a neural network that distinguishes cat and dog pictures, in the worst case that would seem to require a huge network with many nodes/layers (say, the number being a function of the size of the image) rather than the number that seems to work reasonably well in practice (six or some other rather low constant number observed in reality). So the number of dimensions over which the “images” are potentially spread is huge, but it’d seem that in the real world one can rearrange the dog and cat images in a “shape” that then allows for relatively easy disentanglement by the neural network; and these shapes can probably be realized in much lower dimensions (in the example, six).This could explain (for some definition of explain) the observed predictive power of relatively small neural networks.