Indeed, this is all true. But do remember that the kolmogorov-arnold theorem says a 3-layer (n:2
n+1:m) network is a universal continuous approximator (using an unknown neuron transfer function) -- people in the '80s were looking at 3 layer networks as sufficient partly because of that.I have no time to go look at all those sources now, but having dabbled in nets since the late '80s myself, I remember vanishing gradients were sort-of a surprise, because everyone was under the impression that simple backpropagation should just work, and it didn't.
A lot of that early work you refer to was also mostly about linear transfer functions, and though the exact type of non-linearity doesn't matter, some of its properties do - and as I mentioned, sigmoids - which were all the rage in the '80s - are a dead end with the wrong kind of nonlinearity.
Nothing about the structure* of multilayer models is new. But successfully training them - which didn't happen until Schmidhuber and Hinton (depends on who you ask ...) - is relatively new; and that advance is responsible for the term "deep learning".
We do not disagree about the details; but we do seem to disagree about the historical context and narrative.
I have no time to go look at all those sources now, but having dabbled in nets since the late '80s myself, I remember vanishing gradients were sort-of a surprise, because everyone was under the impression that simple backpropagation should just work, and it didn't.
A lot of that early work you refer to was also mostly about linear transfer functions, and though the exact type of non-linearity doesn't matter, some of its properties do - and as I mentioned, sigmoids - which were all the rage in the '80s - are a dead end with the wrong kind of nonlinearity.
Nothing about the structure* of multilayer models is new. But successfully training them - which didn't happen until Schmidhuber and Hinton (depends on who you ask ...) - is relatively new; and that advance is responsible for the term "deep learning".
We do not disagree about the details; but we do seem to disagree about the historical context and narrative.