Comment by oliwaw

>Not necessarily, often derivatives are analytically known in ML.

The focus here is largely on deep neural networks. In this domain, the Hessian cannot be computed and SGD (with minor variants) continues to be the golden standard.