Comment by usernametaken29

When you want to find partial derivatives for a non linear function you need the Hessian matrix as a guarantee that the term itself will be derivable. But in order to find values to put into the Hessian we often rely on heuristics, otherwise you could have the whole trajectory not making progress or swinging wildly. That’s called conditioning. Due to the nature of the underlying geometry some functions are easier and faster to do this for (scalar multiplication comes to mind) than others (this is why soft max is great because it can be rewritten to fit neatly into the Hessian). But of course limiting the families of functions we choose also means limiting the system as a whole, and eliminating a few very non convex cases from our objective altogether … theoretically reducing the capacity of the models