It's because the movement is ascribed to the expansion of space itself, not the individual galaxies. We don't have any reason to believe galaxies are moving in random directions at random speeds (not at scales that explain the redshifts we call the expansion of the universe).
In your explanation, I think we'd expect to see some very distant, very slow-moving galaxies moving toward us. And there may be some very fast-moving galaxies close to us that just started really far away. Objects would be entering our local universe from outside it, and that simply doesn't happen.
In your explanation, I think we'd expect to see some very distant, very slow-moving galaxies moving toward us. And there may be some very fast-moving galaxies close to us that just started really far away. Objects would be entering our local universe from outside it, and that simply doesn't happen.