That’s not wrong but does misrepresent history.
For many years a 3 layer network (1 input, 1 hidden, 1 output) was the standard and considered sufficient, thanks to several relevant but non constructive approximation theorems (mostly one due to Kolmogorov and one due to Cybenko, if I am not mistaken).
They were also considered practically required because everyone was using sigmoids which have a vanishing gradient problem.
Several things were needed to break away to where we are now: unbounded functions (like ReLU) to avoid vanishing moments; a lot more layers; a lot more parameters and compute power.
When Schmidhuber (and later Hinton) showed that many-layer nets work well, that was non trivial and almost revolutionary.
It is now trivial and all nets are “deep”. But that wasn’t the case when the breakthroughs were made, and the terminology stuck.
For many years a 3 layer network (1 input, 1 hidden, 1 output) was the standard and considered sufficient, thanks to several relevant but non constructive approximation theorems (mostly one due to Kolmogorov and one due to Cybenko, if I am not mistaken).
They were also considered practically required because everyone was using sigmoids which have a vanishing gradient problem.
Several things were needed to break away to where we are now: unbounded functions (like ReLU) to avoid vanishing moments; a lot more layers; a lot more parameters and compute power.
When Schmidhuber (and later Hinton) showed that many-layer nets work well, that was non trivial and almost revolutionary.
It is now trivial and all nets are “deep”. But that wasn’t the case when the breakthroughs were made, and the terminology stuck.