My experience running several ssh servers on uncommon nonstandard ports for over 10 years has been that it has eliminated all ssh brute forcing. I don’t even bother with fail2ban. I probably should though, just in case.

Also, PSA: if you use fail2ban, don’t try tab completing rsync commands without using controlmaster or you will lock yourself out.