If it is just you on your server and the only access from outside your network is SSHing in front the VPN? You’re good. Especially if it’s just you on your network/VPN.

If there are services that others utilize, you need a firewall. Can’t trust other people’s devices to not drag in malware.