These are nice animations. However I've always hesitated to get too enamored with these simple 2D visualizations of gradient descent, because one of the strange takeaways from deep learning is that behavior in high dimensions is very different from behavior in low dimensions.
In a 2D problem with many local minima, like the "eggholder functions" [1], gradient descent will be hopeless. But neural net optimization in high dimensions really is a similar situation with many local minima, except gradient descent does great.
Gradient descent in high dimensions also seems to have the ability to "step over" areas of high loss, which you can see by looking at the loss of a linear interpolation between weights at successive steps of gradient descent. This, again, seems like extremely strange behavior with no low-dimensional analogue.
In a 2D problem with many local minima, like the "eggholder functions" [1], gradient descent will be hopeless. But neural net optimization in high dimensions really is a similar situation with many local minima, except gradient descent does great.
Gradient descent in high dimensions also seems to have the ability to "step over" areas of high loss, which you can see by looking at the loss of a linear interpolation between weights at successive steps of gradient descent. This, again, seems like extremely strange behavior with no low-dimensional analogue.
[1] https://www.sfu.ca/~ssurjano/egg.html