Comment by albertzeyer

Note that such a 2D parameter space gives often the wrong intuition when thinking about applying gradient descent on high-dimensional parameter space.

Also, mini-batch stochastic gradient descent behaves more stochastic than just gradient descent.