> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten.
Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.
> In their new proof, the pair show that overparameterization is necessary for a network to be robust.
What is important to note here is that many of papers this paper cites prove or show this result in certain network architectures. This paper adds universality.
> The proof is very elementary — no heavy math, and it says something very general
The most elementary part was clever use of Hoeffding's inequality. Some people are really fast readers haha.
I don't even know how you pick up the fact that isoperimetry holds in manifold settings with positive curvature while also playing with all those norms and inequalities. A few years ago I mentioned on here all the maths that I knew or wanted to know to read more papers, and others critiqued that the list was too long. Well, this is why!
Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.
> In their new proof, the pair show that overparameterization is necessary for a network to be robust.
What is important to note here is that many of papers this paper cites prove or show this result in certain network architectures. This paper adds universality.
> The proof is very elementary — no heavy math, and it says something very general
The most elementary part was clever use of Hoeffding's inequality. Some people are really fast readers haha.
I don't even know how you pick up the fact that isoperimetry holds in manifold settings with positive curvature while also playing with all those norms and inequalities. A few years ago I mentioned on here all the maths that I knew or wanted to know to read more papers, and others critiqued that the list was too long. Well, this is why!