Security through obscurity is not the solution, though. Sooner or later someone does a port scan and finds that port. I've found that it's more efficient just to whitelist the IPs or subnets you might need and denying the rest.
Edit: I didn't mean changing the port is a bad thing. You can and should do it, and it will help a bit - just that it shouldn't be the only thing to rely on for your SSH security.
Edit: I didn't mean changing the port is a bad thing. You can and should do it, and it will help a bit - just that it shouldn't be the only thing to rely on for your SSH security.