Comment by credit_guy

> But the variance does not cancel out.

They are multiplied with the derivative in that direction, which is zero.