Some docker containers need manual start after host reboot

Submitted ⁨⁨4⁩ ⁨weeks⁩ ago⁩ by ⁨OneCardboardBox@lemmy.sdf.org⁩ to ⁨selfhosted@lemmy.world⁩

I generally let my server do its thing, but I run into an issue consistently when I install system updates and then reboot: Some docker containers come online, while others need to be started manually. All containers were running before the system shut down.

My containers are managed with docker compose.
Their compose files have restart: always
It’s not always the same containers that fail to come online
Some of them depend on an NFS mount point being ready on the host, but not all

Host is running Ubuntu Noble

Most of these containers were migrated from my previous server, and this issue never manifested.

I wonder if anyone has ideas for what to look for?

source

Comments

Sort:hotnew top

WhyJiffie@sh.itjust.works ⁨4⁩ ⁨weeks⁩ ago
I have recently discovered what was causing this to me for years. It was IP specific port bindings. Ports of a few containers were only bound for the LAN IP of the system, but if DHCP couldn’t obtain an IP until the Docker service started its startup, then those containers couldn’t be started at all, and Docker in it’s wisdom won’t bother with retrying.

The reasons to move my compose stacks to separate systemd services are counting.

source
CameronDev@programming.dev ⁨4⁩ ⁨weeks⁩ ago
Can you make your docker service start after the NFS Mount to rule that out?

A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container which doesn’t start at all from going into a restart loop.

docs.docker.com/…/start-containers-automatically/…

If your containers are crashing before the 10 timeout, then they won’t restart.

source
- OneCardboardBox@lemmy.sdf.org ⁨3⁩ ⁨weeks⁩ ago
  Yep, the problem was that docker started before the NFS mount. Adding the dependency to my systemd docker unit did the trick!
  
  source
  - CameronDev@programming.dev ⁨3⁩ ⁨weeks⁩ ago
    Excellent, thanks for the update!
    
    source
mbirth@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago
Put that mount point into the compose file(s). You can define volumes with type nfs and basically have Docker-Compose manage the mounts.

source
just_another_person@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
Add a healthcheck to test the mount directory. If it’s deemed unhealthy, the container should restart until it passes. If your NFS mount is automounting properly, you should fix that though.

source