Just use wireguard as VPN and bind ssh only to that interface. You loose public access but I couldn’t think of a reason why I want other devices than my own to connect anyway.