How i do it:

• Wireguard for VPN endpoint on the pi and device that I have root on, secure, fast to setup and doesn’t add a lot of overhead

• For access outside of VPN:

You can manually allow trusted IP to access the service in your firewall which nullify surface of attack if done perfectly but is really an hassle to setup and maintain. I’m looking to setup Keycloack for a strong pre-auth that I can share between services and that is also lightweight (Authentik is not lightweight, Authelia seems to be i’d like to try it aswell) This coupled with firewall rules and/or fail2ban like service should be more than enough for a private server I think.