Yes, reverse proxy, but you don’t want to publish on different IP addresses. Your services should bind to one IP, different ports, and the reverse proxy accepts it all on 443 and routes it based on the host header.

I use traefik for this, set labels in the docker compose and it Just Works. It also gets certs for me based on the acme DNS challenge. Some people use caddy instead of traefik and they seem happy with it.