That’s always a danger.

You could host the server on its own vlan and use tailscale to tunnel from a VPS to your server. This way your IP isn’t exposed to the public. You can also add crowdsec or similar on the VPS.

I do the same thing with traefik on my dirt cheap VPS.