Comment by alerighi

You can maintain a .ssh/config file with all the port that you used (and backup it) or use always the same custom port.

Anyway, it is useless but it can avoid all sort of bots trying out common login to your SSH server, that consumes resources (even if minimal) and fills your logs.