Comment on Backup for important files/pictures?
CoyoteFacts@lemmy.ca 3 weeks ago
I use a 48TB ZFS RAIDZ2 pool to maintain data integrity locally and keep rolling ZFS snapshots with sanoid so that data recovery to any point within the last month is possible. Then I use borgmatic (borg) to sync the important data (~1TB) to a Servarica VPS (Polar Bear plan, which works out to be cheaper than Backblaze B2 costs for my purposes). The Servarica server really sucks in terms of CPU, and it’s quite sluggish, but it’s enough for daily backups. I also self-host healthchecks.io on a free Fly.io VPS thing (not sure if they offer this anymore) to make sure the backups are actually happening successfully. I run the same borg configuration to back up to a plain non-redundant disk locally.
The downside of this setup is that I’m only truly backing up a fraction of my pool, but most of my pool is stuff that I can redownload and set up again in the event of a house fire. I also run a daily script to dump a lot of metadata about my systems and pool, like directory listings of my media folders and installed programs/etc, which means that even though the data might be lost, I have a map of what I need to grab again.
CmdrShepard49@sh.itjust.works 3 weeks ago
How big are these ZFS snapshots compared to the stored data size? 1:1?
CoyoteFacts@lemmy.ca 3 weeks ago
Snapshots basically put a pin in all the data at that moment and say “these blocks are not going to be physically deleted as long as I exist”, so the “additional” data use of the snapshots is equal to the data contained within the snapshot that doesn’t exist at the current moment. I.e., if I have two 50GB files, take a snapshot, and delete one, I will still have 100GB physical disk usage. I can also take 400 more snapshots and disk usage will remain at 100GB, as the snapshots are just virtual. Then I can either bring that deleted file back from the snapshot, or I can delete the snapshot itself and my disk usage will adjust to the “true” 50GB as the snapshot releases its hold on the blocks.
What sanoid and other snapshot managers do is they repeatedly take snapshots at specified intervals and delete old snapshots past a certain date, which in practice results in a “rolling” system where you can access snapshots from e.g. every hour in the past month. Then once a snapshot becomes older than a month, sanoid will auto-delete it and free up the disk space that it’s holding onto. The exact settings are all configurable to what you’re comfortable with in terms of trading additional physical disk usage of potential “dead” data for the convenience of being able to resurrect that data for a certain amount of time.
I really like the “data comet” visual from this Ars Technica article.