I have always had trouble whenever I tried mounting files, and it seems that you’re getting the same error, “not a directory”.

Stick to mounting directories and you should be fine

/home/Joe/container/caddy:/etc/caddy