Comment by jordigh

I need to write something about conjugate gradients. The first thing we did in my optimisation class at CIMAT was gradient descent and showed how it was so slow compared to everything else. In practical applications, there's way too much magic in choosing the so-called "learning rate" (known as "step size" in numerical optimisation). Even choosing an optimal step size per iteration (by doing a line search), gradient descent is still slow. Conjugate gradients are a bit more difficult to understand, but they're not that hard to implement if you just blindly copy a few formulae, and they are a dramatic speedup.

All of these blog posts about gradient descent feel like everyone keeps going on about what a great algorithm bubblesort is because it's so easy to understand and implement.