Key only authentication is your easiest and safest bet. If you want something a little more elegant, look at setting up port knocking if this is a 24/7 service exposed to the public internet. Add a brute-force monitor like fail2ban or crowdsec as an added layer of protection if you want.