Comment by duvenaud

This is simply wrong. Backprop has the same asymptotic time complexity as forward.