Comment by cracker_jacks

I believe what you are describing is grid search, not gradient descent. Gradient descent is not an exhaustive search.