Comment by mochomocha

Line search is expensive. It costs you 1 pass over the data just for function evaluation. Also, there's only theoretical guarantees in convex cases (and even in these cases, details get murky if you look at L-BFGS and its treatment in Nocedal optimization book for example). Most modern neural networks are optimized over non-convex loss functions and through SGD optimizers which prove to be extremely effective because they make quick progress for super cheap (mandatory plug to this classic NIPS paper: https://papers.nips.cc/paper/3323-the-tradeoffs-of-large-sca...).