This is primarily in regard to deep neural nets, where second-order methods are too expensive (O(p^2) in parameters, where p is on the order of millions).
I'm not sure conjugate gradient methods are used much, due to the non-convexity of the merit functions, and hard constraints aren't used much either.
I'm not sure conjugate gradient methods are used much, due to the non-convexity of the merit functions, and hard constraints aren't used much either.