Does SSH have to be your only way? Could you deploy something like Tailscale? Can you restrict the allowed IP ranges on SSH with a firewall rule?