Comment by maxerize

Gradient descent is neither complex nor ugly, it's the most immediate and straightforward way of searching for extrema. Iterative methods in general might be your only shot if the problem is not linear and linearisation is not an option. From a practical point of view, iterative methods in general allow you to terminate on your own terms, whereas many direct methods don't have a useful representation at hand to return prematurely.