Docker Backup Stratagy

Submitted ⁨⁨12⁩ ⁨hours⁩ ago⁩ by ⁨njordomir@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

Hello Self-Hosters,

What is the best practice for backing up data from docker as a self-hoster looking for ease of maintenance and foolproof backups. (pick only one :D )

Assume directories with user data are mapped to a NAS share via NFS and backups are handled separately.

My bigger concern here is how do you handle all the other stuff that is stored locally on the server, like caches, databases, etc.

Is it better to run #cp /var/lib/docker/volumes/* /backupLocation every once in a while, or is it preferable to define mountpoints for everything inside of /home/user/Containers and then use a script to sync it to wherever you keep backups? What pros and cons have you seen or experienced with these approaches?
How do you test your backups? I’m thinking about digging up an old PC to use to test backups. I assume I can just edit the ip addresses in the docker compose, mount my NFS dirs, and failover to see if it runs.
I started documenting my system in my notes and making a checklist for what I need to backup and where it’s stored. Currently trying to figure out if I want to move some directories for consistency. Can I just do docker-compose down edit the mountpoints in docker-compose.yml and run docker-compose up to get a working system?

source

Comments

Sort:hotnew top

lazynooblet@lazysoci.al ⁨8⁩ ⁨hours⁩ ago
I’m lucky enough to run a business that needs a datacenter presence. So most my home-lab (including Lemmy) is actually hosted on a Dell PowerEdge R740xd in the DC. I can then use the small rack I have at home as off-site backups and some local services.

I treat the entirety of /var/lib/docker as expendable. When creating containers, I make sure any persistent data is mounted from a directory made just to host the persistent data. It means docker compose down --rmi all --volumes isn’t destructive.

When a container needs a database, I make sure to add an extra read-only user. And all databases have their container and persistent volume directory named so scripts can identify them.

The backup strategy is then to backup all non-database persistent directories and dump all SQL databases, including permissions and user accounts. This gets run 4 times a day and the backup target is an NFS share elsewhere.

This is on top of daily backuppc backups of critical folders, automated Proxmox snapshots for docker hosts every 20 minutes, daily VM backups via Proxmox Backup Server and replication to another PBS at home.

I also try and use S3 where possible (seafile and lemmy are the 2 main uses) which is hosted in a container on a Synology RS2423RP+. Synology HyperBackup then performs a backup overnight to the Synology RS822+ I have at home.

Years ago I fucked up, didn’t have backups, and lost all the photos of my sons early years. Backups are super important.

source
- hazl@lemmy.blahaj.zone ⁨6⁩ ⁨hours⁩ ago
  That’s a heart breaking way to learn your lesson about backups. It must have taken a long time to accept.
  
  source
  - pory@lemmy.world ⁨4⁩ ⁨hours⁩ ago
    Everyone I know that actually keeps backups has the same kind of story. It’s sad that no matter how many other people talk about keeping backups, it always takes a tragic loss like this to get people to buy hardware/subscriptions.
    
    source
    -> View More Comments
catloaf@lemm.ee ⁨4⁩ ⁨hours⁩ ago
Bind mounts. I’ve never bothered to figure out named volumes, since I often work with the contents outside Docker. Then I just back up the whole proxmox VM. (Yes I’m aware proxmox supports containers, no I don’t plan to convert, that’s more time and effort for no meaningful gain to me.)

You can restore that backup to a new VM. I just make sure it boots and I can access the files. Turn off networking before you boot it so that it doesn’t cause conflicts.

source
- jacksilver@lemmy.world ⁨1⁩ ⁨hour⁩ ago
  Yep, bind mount the data and config directories and back those up. You can test a backup by spinning up a new image with the data/config directories.
  
  This is both easy and generally the recommended thing I’ve seen for many services.
  
  The only thing that could cause issues is breaking changes caused by the docker images themselves, but that’s an issue regardless of backup strategy.
  
  source
Sinirlan@lemmy.world ⁨11⁩ ⁨hours⁩ ago
I just took line of least effort, all my docker containers are hosted on dedicated VM in proxmox, so I just backup entire VM on weekly basis to my NAS. Already had to restore it once when I was replacing SSD in proxmox host, worked like a charm.

source
- njordomir@lemmy.world ⁨3⁩ ⁨hours⁩ ago
  I miss this from cloud hosting. It’s helpful to be able to save, clone, or do whatever with the current machine state and easily just flash back to where you were if you mess something up. Might be too much to set up for my current homelab though. My server does have btrfs snapshots of everything directly in grub which has let me roll back a few big screwups here and there.
  
  source
- dataprolet@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago
  While that’s an easy solution it makes it impossible or rather difficult to restore single containers and/or files.
  
  source
  - yournamehere@lemm.ee ⁨2⁩ ⁨hours⁩ ago
    bs
    
    source
  - tofu@lemmy.nocturnal.garden ⁨7⁩ ⁨hours⁩ ago
    It’s not that difficult. You can create a second VM from the backup with a few clicks and move the necessary data with scp.
    
    source
greatley@ani.social ⁨11⁩ ⁨hours⁩ ago
I don’t know if it’s best practice but I mount everything to a specific folder instead of volumes. I also just stop all the containers before a backup instead of database dumping. Then just do an encrypted B2 backup using Restic. So far I had to restore twice in six years of running this system and assuming the folder path is the same the only thing after downloading all the data I had to do was

docker-compose up -d
source
- lorentz@feddit.it ⁨3⁩ ⁨hours⁩ ago
  I’d say that the most important takeover of this approach is to stop all the containers before the backup. Some applications (like databases) are extremely sensitive to data corruption. If you simply ´cp´ while they are running you may copy files of the same program at different point in time and get a corrupted backup. It is also important mentioning that a backup is good only if you verify that you can restore it. There are so many issues you can discover the first time you recover a backup, you want to be sure you discover them when you still have the original data.
  
  source
plumbercraic@lemmy.sdf.org ⁨11⁩ ⁨hours⁩ ago
I wouldn’t backup the volumes directly. Better to use the mount points as you suggest then back up those mounted directories. If it’s a database that usually needs to have its records exported into a backup friendly format. Typically I will do a db dump from a cron job in the host system to summon a script inside a container which writes to a mounted dir which is the thing that I back up.

source
kewjo@lemmy.world ⁨10⁩ ⁨hours⁩ ago
caches are never really a concern to me they will regen are the fact, from your description i would work more about db, this is dependent though in what you’re using and what you are storing. if the concern is having the same system intact then my primary concern would be backing up any config file you have. in cases of failure you mainly want to protect against data loss, if it takes time to regenerate cache/db that’s time well spent for simplicity of actively maintaining your system

source
confusedpuppy@lemmy.dbzer0.com ⁨7⁩ ⁨hours⁩ ago
I created my own script/tool using rsync to handle backups and transferring data.

My needs are quite smaller with just a computer and two Raspberry Pi’s but I found rsync to be really useful overall.

My backup strategy is to make a complete backup on the local device (Computer / RPi4 / RPi5) then copy all those backups to a Storage partition on my computer, then make a whole backup from the partition to an externally attached SSD.

The RPi’s both use docker/podman containers so I make sure any persistent data is in mounted directories. I usually stop all containers before performing a backup, especially things with databases.

Everything in the docker containers is either hit or miss when it comes to restoring. The simple docker images restore as it they were untouched and will launch like nothing happened. I have a PieFed instance that must be rebuilt after restoring a backup. Since PieFed’s persistent data is in mount points, everything works perfectly after a fresh build.

I can send a link to my rsync tool if that’s any interest to anyone. I’ve found it super useful for backups and minimizes so much headache for myself when it comes to transferring files between different network connected devices.

source
dataprolet@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago
I use Borg to backup the default volume directory and my compose files. If you’re interested I can share my backup script.

source
bdonvr@thelemmy.club ⁨8⁩ ⁨hours⁩ ago
BTRBK for easy BTRFS snapshots to an external drive.

Rsync to also upload those to B2 compatible storage

source