Why use Caddy at all then? Tailscale would work just as well, with the added bonus that you could access it outside your LAN without exposing it to the clearnet.