Comment by cuchoi

"Least squares" means that the overall solution minimizes the sum of the squares. In some cases you can get to that result analytically (I imagine that this is what you are referring to as "least squares"). In other cases you can't do it analytically. Also, sometimes it just a lot faster to use SGD or other algorithms.