I think VPN is the proper way to go about this, but another method is to do port knocking with fkwnop so your SSH port won’t respond until the host receives a magic packet.