Whats your setup? Stuff dockerized? Looks like routing/iptables are set incorrectly (depending on your setup).

But my solution would be using a local DNS VPN like personalDNSfilter, exposing your services over the VPS to the net and connecting over https to them. For local access at home you can use split-brain-dns (hope thats the correct word [example.net = 8.8.8.8, at home it resolutes to 192.168.x.y]). With that you can just walk around and use every service with the best latency and bandwidth, without manually deactivating the VPN.