but I’d like to give Nginx Proxy Manager a try, it seems easier to manage stuff not in docker.

NPM is pretty agnostic. If it receives a request for a specific address and port combination it just forwards the traffic to another specific address and port combination. This can be a docker container, but also can be a physical machine or any random URL.

It also has Let’s Encrypt included (but that should be a no-brainer).