Caddy does not need 80 and 443.

By default and all measurable expectation it does. Unless you can’t use privileged HTTP/HTTPS ports, there’s no real reason to use unprivileged ports.

Besides, op doesn’t mention having problems with ports

OP said he was having issues, and this is a common issue I’ve had. Since he was non-descript as to what the issues were, it’s really not stupid to mention it.