Network loss after 24hrs on Docker LXC

Submitted ⁨⁨2⁩ ⁨months⁩ ago⁩ by ⁨OminousOrange@lemmy.ca⁩ to ⁨selfhosted@lemmy.world⁩

Fine folks of c/selfhosted, I’ve got a Docker LXC (Debian) running in Proxmox that loses its local network connection 24 hours after boot. It’s remedied with a LXC restart. I am still able to access the console through Proxmox when this happens, but all running services (docker ps still says they’re running) are inaccessible on the network. Any recommendations for an inexperienced selfhoster like myself to keep this thing up for more than 24 hours?

Tried:

Pruning everything from Docker in case it was a remnant of an old container or something.
Confirming network config on the router wasn’t breaking anything.
Checked there were no cron tasks doing funky things.

I did have a Watchtower container running on it recently, but have since removed it. It being a 24 hr thing got me thinking that was the only thing that would really cause an event at the 24 hr post start mark, and it started about that same time I removed Watchtower (intending to do manual updates because immich).

…and of course, any fix needs 24 hours to confirm it actually worked.

A forum post I found asked for the output of ip a and ip r, see below. Notable difference on ip r missing the link to the gateway after disconnecting.

Running normally:

ip a

spoiler

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0@if53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 26:7b:1d:04:0f:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.1.104/24 brd 192.168.1.255 scope global dynamic eth0 valid_lft 85189sec preferred_lft 85189sec 3: br-04287dfab180: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:d4:b3:85:4d brd ff:ff:ff:ff:ff:ff inet 172.24.0.1/16 brd 172.24.255.255 scope global br-04287dfab180 valid_lft forever preferred_lft forever 4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:4a:0e:28:49 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 5: br-dc3db6eba7fb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:b2:84:cc:c5 brd ff:ff:ff:ff:ff:ff inet 172.25.0.1/16 brd 172.25.255.255 scope global br-dc3db6eba7fb valid_lft forever preferred_lft forever 7: veth15a2027@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 1a:c6:8a:c7:38:88 brd ff:ff:ff:ff:ff:ff link-netnsid 4 9: veth88a187b@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether ea:20:b7:0c:fe:de brd ff:ff:ff:ff:ff:ff link-netnsid 1 11: vetha468b68@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether e2:06:5d:ea:2b:94 brd ff:ff:ff:ff:ff:ff link-netnsid 2 13: veth5104bb2@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 3e:44:8c:14:05:ec brd ff:ff:ff:ff:ff:ff link-netnsid 6 15: vethe33371c@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-04287dfab180 state UP group default link/ether 0e:29:60:01:c8:91 brd ff:ff:ff:ff:ff:ff link-netnsid 5 17: veth1abc5f6@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 46:63:4d:8c:93:3d brd ff:ff:ff:ff:ff:ff link-netnsid 3 19: veth5649f31@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 36:90:9e:f8:79:fb brd ff:ff:ff:ff:ff:ff link-netnsid 7 21: vethff1c73b@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 0a:8e:85:4f:fe:56 brd ff:ff:ff:ff:ff:ff link-netnsid 9 23: vethc5655a4@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether da:90:f1:76:7b:6e brd ff:ff:ff:ff:ff:ff link-netnsid 8

ip r

spoiler

default via 192.168.1.1 dev eth0 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.24.0.0/16 dev br-04287dfab180 proto kernel scope link src 172.24.0.1 172.25.0.0/16 dev br-dc3db6eba7fb proto kernel scope link src 172.25.0.1 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.104

Disconnected from local network:

ip a

spoiler

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0@if49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 26:7b:1d:04:0f:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 3: br-04287dfab180: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ad:93:5d:1c brd ff:ff:ff:ff:ff:ff inet 172.24.0.1/16 brd 172.24.255.255 scope global br-04287dfab180 valid_lft forever preferred_lft forever 4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:bf:99:74:1a brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 5: br-dc3db6eba7fb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:fb:44:63:30 brd ff:ff:ff:ff:ff:ff inet 172.25.0.1/16 brd 172.25.255.255 scope global br-dc3db6eba7fb valid_lft forever preferred_lft forever 7: veth050be68@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 72:80:37:e9:40:55 brd ff:ff:ff:ff:ff:ff link-netnsid 1 9: vethc3d3375@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 32:75:4a:55:4b:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 2 11: vethe58fc45@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 0a:2d:6a:5e:d9:bf brd ff:ff:ff:ff:ff:ff link-netnsid 3 13: veth3b7b2de@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-04287dfab180 state UP group default link/ether 86:ab:c6:51:a9:3d brd ff:ff:ff:ff:ff:ff link-netnsid 4 15: veth1965980@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether b2:3d:1c:4f:1e:5c brd ff:ff:ff:ff:ff:ff link-netnsid 6 17: veth3d92cbb@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 02:ac:5b:56:e4:33 brd ff:ff:ff:ff:ff:ff link-netnsid 5 19: vethf691ec9@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 3e:dd:e3:7b:df:79 brd ff:ff:ff:ff:ff:ff link-netnsid 7 21: veth53d911c@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 72:ac:8e:a3:ab:97 brd ff:ff:ff:ff:ff:ff link-netnsid 9 23: vethe8ec657@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dc3db6eba7fb state UP group default link/ether 06:c4:32:b4:a6:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 8

ip r

spoiler

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.24.0.0/16 dev br-04287dfab180 proto kernel scope link src 172.24.0.1 172.25.0.0/16 dev br-dc3db6eba7fb proto kernel scope link src 172.25.0.1

source

Comments

Sort:hotnew top

lemmyreader@lemmy.ml ⁨2⁩ ⁨months⁩ ago
Are you running Docker from within Proxmox or next besides Proxmox ? I am not familiar with Proxmox these days, but I do know that standalone Docker + lxc via lxd can make the network of the lxc containers time out due to Docker iptables setup. Here’s a blog post about that : seanblanchfield.com/…/bridge-networking-in-lxd

source
- OminousOrange@lemmy.ca ⁨2⁩ ⁨months⁩ ago
  Docker is installed on a Debian container with Proxmox as the hypervisor. I believe as far as Docker knows, it’s just running on normal Debian. The Debian LXC has its own local ip.
  
  I’ll take a look at those resources though, thanks.
  
  source
emptiestplace@lemmy.ml ⁨2⁩ ⁨months⁩ ago
Why are you doing this? Unless you have a good reason, you should probably either run Docker in a vm, or use something other than Proxmox where you can just install Docker on the host system.

source
- skittlebrau@lemmy.world ⁨2⁩ ⁨months⁩ ago
  Not OP, but I run Docker in LXC because my Proxmox host is an Intel NUC and I only have one graphics card (integrated).
  
  I don’t want to passthrough the iGPU to a VM because then I lose video output for the host. I also don’t want to use SR-IOV for iGPU because it’s buggy and results in garbled output for HDR content. That’s why, in my case, Docker in LXC makes sense.
  
  source
  - OminousOrange@lemmy.ca ⁨2⁩ ⁨months⁩ ago
    This is mostly my reasoning too. I’ve got a bit more juice than a NUC, but I prefer the way resources are managed with an LXC for the certain apps that I run. I still have VMs for other things, like HAOS and a BlueIris NVR. It’s only a local homelab with no external users so avoiding additional complexity is often in my best interest.
    
    source
  - emptiestplace@lemmy.ml ⁨2⁩ ⁨months⁩ ago
    Why not just use LXC though?
    
    source
    -> View More Comments
- thirdBreakfast@lemmy.world ⁨2⁩ ⁨months⁩ ago
  My ‘good reason’ is just that it’s super convenient - for backups and painlessly moving apps around between nodes with all their data.
  
  I would run plain LXCs if people nicely packaged up their web apps as LXC templates and made them available on LXCHub for me to run with lxc compose up, but they generally don’t.
  
  I guess another alternate future would be if Proxmox added docker container supervision to their web interface, but you’re still not going to have the self-contained neat snapshot system that includes the data.
  
  In theory you should be able to convert an OCI container layer by layer into an LXC, so I bet there’s projects out there that attempt this.
  
  Image
  
  source
  - markstos@lemmy.world ⁨2⁩ ⁨months⁩ ago
    It’s convenient when it works, but with three different containerization technologies, it’s harder to debug when Proxmox+LXC+Docker fails. Even running Docker in parallel to LXC rather than nested would be simpler.
    
    source
- pyrosis@lemmy.world ⁨2⁩ ⁨months⁩ ago
  Usually this comes down to resource and energy efficiency. While a vm works perfectly fine you will find you can share video and storage resources in efficient ways with lxc.
  
  For example you can directly pass a zfs dataset into a lxc with a simple lxc.mount.entry:
  
  This would allow you to configure options like cluster size, atime, compression algorithm, xattr, etc… without much overhead.
  
  It’s also nice to know you can share your GPU with multiple lxc without it being locked into a single vm.
  
  source
- OminousOrange@lemmy.ca ⁨2⁩ ⁨months⁩ ago
  Why would one prefer a VM over an LXC for Docker?
  
  source
  - emptiestplace@lemmy.ml ⁨2⁩ ⁨months⁩ ago
    
    Kernel isolation
    
    HA
    
    Stability
    
    It works
    
    source
    -> View More Comments
Decronym@lemmy.decronym.xyz ⁨2⁩ ⁨months⁩ ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters

HA Home Assistant automation software

~ High Availability

LXC Linux Containers

NUC Next Unit of Computing brand of Intel small computers

[Thread #685 for this sub, first seen 17th Apr 2024, 07:05] [FAQ] [Full list] [Contact] [Source code]

source
thirdBreakfast@lemmy.world ⁨2⁩ ⁨months⁩ ago
No answer, but just to say I run most of my services with this setup - Docker in a Debian LXC under Proxmox, and don’t have this issue. The containers are ‘privileged’, and I have ‘nesting’ ticked on, but apart from that all defaults.

source
- OminousOrange@lemmy.ca ⁨2⁩ ⁨months⁩ ago
  I might have found the issue, see updates above. I have a separate Docker LXC that was behaving normally, so was good to cross-check with.
  
  source

Fewer Letters	More Letters
HA	Home Assistant automation software
~	High Availability
LXC	Linux Containers
NUC	Next Unit of Computing brand of Intel small computers