Disable the firewall if you can to check if that’s the issue, then do a tcpdump using root with the port. Do tcpdump inside the container too and compare what you see to the docker environment.

Is caddy-caddy really the correct image?

Try with this command, it’s the minimal setup that works by default (on my machine): podman run -p 0.0.0.0:5050:80 docker.io/library/caddy:latest