Yes, I meant "linear least squares" which is "(X.T X)^{-1} X.T y".
> Also, sometimes it just a lot faster to use SGD or other algorithms.
Right, that's what I thought. Is this because essentially optimization will be bounded by the number of of epochs but linear least squares will be bounded by matrix operations (so scale with N). Which means, if you can solve the problem in small number of epochs (say, 200) and N is very large then SGD will be faster. Is this correct?
EDIT: Obviously, I don't think all regression problems can be solved this way but in this blog post their loss function can be solved by linear least squares. If you solve the optimization problem analytically, you'll get "(X.T X)^{-1} X.T y".
> Also, sometimes it just a lot faster to use SGD or other algorithms.
Right, that's what I thought. Is this because essentially optimization will be bounded by the number of of epochs but linear least squares will be bounded by matrix operations (so scale with N). Which means, if you can solve the problem in small number of epochs (say, 200) and N is very large then SGD will be faster. Is this correct?
EDIT: Obviously, I don't think all regression problems can be solved this way but in this blog post their loss function can be solved by linear least squares. If you solve the optimization problem analytically, you'll get "(X.T X)^{-1} X.T y".