When add a remote server in the following way, the port of git service is 22.
git remote add origin user@address:/path/to/repo.git
You can easily change that by adding like this:
git remote add origin ssh://user@address:port/path/to/repo.git
Or you can just change the config file in .git folder.