Comment by untrimmed

So all the classic optimization theory about staying in the stable region is basically what deep learning doesn't do. The model literally learns by becoming unstable, oscillating, and then using that energy to self-correct.

The chaos is the point. What a crazy, beautiful mess.