Comment by partykid92

quasi newton methods are not square in the dimension of the problem (think limited memory L-BFGS), and can be run in linear time. In my experience, however, they're 2-3 times slower than regular methods like ADAM.