Comment on Incremental backups to optical media: tar, dar, or something else?
traches@sh.itjust.works 2 days agoI’m using standard BD-DLs. M-Disks are almost triple the price, and this project is already too costly. I’m not looking for centuries of longevity, I’m using optical media because it’s read-only once written. I read that properly stored Blu-Rays should be good for 10 or 20 years, which is good enough for me. I’ll make another copy when the read errors start getting bad.
Copying files directly would work, but my library is real big and that sounds tedious. I have photos going back to the 80s and curating, tagging, and editing them is an ongoing job. (This data is saved in XMP sidecars alongside the original photos). I also won’t be encrypting or compressing them for the same reasons you mentioned.
For me, the benefit of the archive tool is to automatically split it up into disk-sized chunks. That and to automatically detect changes and save a new version; your first key doesn’t hold true for this dataset. You’re right though, I’m sacrificing accessibility for the rest of the family. I’m hoping to address this with thorough documentation and static binaries on every disk.
sxan@midwest.social 21 hours ago
The densities I’m seeing on M-Discs - 100GB, $5 per, a couple years ago - seemed acceptable to me. $50 for a TB? How big is your archive? Mine still fits in a 2TB disk.
I mean, putting it in an archive isn’t going to make it any smaller. Compression on even lossless compressed images doesn’t often help.
And we’re talking about 100GB discs. Is squeezing that last 10MB out of the disk by splitting an image across two disks worth it?
The metadata is a different matter. I’d have to think about how to handle the sidecar data… but that you could almost keep on a DVD-RW, because there’s no way that’s going to be anywhere near as large as the photos themselves. Is your photo editor DB bigger than 4GB?
I never change the originals. When I tag and edit, that information is kept separate from the source images - so I never have multiple versions of pictures, unless I export them for printing, or something, and those are ephemeral and can be re-exported by the editor with the original and the sidecar. Music, and photos, I always keep the originals isolated from the application.
This is good, though; it’s helping me clarify how I want to archive this stuff. Right now mine is just backed up on multiple disks and once in B2, but I’ve been thinking about how to archive for long term storage.
I think in going to go the M-Disc route, with sidecar data on SSD and backed up to BluRay RW. The trick will be letting DarkTable know that the source images are on different media, but I’m pretty sure I saw an option for that. For sure, we’re not the first people to approach this problem.
The whole static binary thing - I’m going that route with an encrypted share for financial and account info, in case I die, but that’s another topic.
traches@sh.itjust.works 13 hours ago
Where I live (not the US) I’m seeing closer to $240 per TB for M-disc. My whole archive is just a bit over 2TB, though I’m also including exported jpgs in case I can’t get a working copy of darktable that can render my edits. It’s set to save xmp sidecars on edit so I don’t bother with backing up the database.
I mostly wanted a tool to divide up the images into disk-sized chunks, and to automatically track changes to existing files, such as sidecar edits or new photos. I’m now seeing I can do both of those and still get files directly on the disk, so that’s what I’ll be doing.
I’d be careful with using SSDs for long term, offline storage. I hear they lose data if not powered for a long time. IMO metadata is small enough to just save a new copy when it changes
sxan@midwest.social 11 hours ago
It’d be more space efficient to store a COW2 of Linux with a minimum desktop and basically only DarkTable on it. The VM format hasn’t changed in decades.
Shoot. A bootable disc containing Linux and the software you need to access the images, and on a separate track, a COW2 image of the same, and on a third, just DarkTable. Best case, you pop in the drive & run DarkTable. Or, you fire up a VM with the images. Worst case, boot into linux. This may be the way I go, although - again - the source images are the important part.
What I meant was, keep the master sidecar on SSD for regular use, and back it up occasionally to a RW disc. Probably with a simply cp -r to a directory with a date. This works for me because my sources don’t change, except to add data, which is usually stored in date directories anyway.
You’re also wanting to archive the exported files, and sometimes those change? Surely, this is much less data? Of you’re like me, I’ll shoot 128xB and end up using a tiny fraction of the shots. I’m not sure what I’d do for that - probably BD-RW. The longevity isn’t great, but it’s by definition mutable data, and in any case the most recent version can be easily enough regenerated as long as I have the sidecar and source image secured.
Burning the sidecar to disk is less about storage and more about backup, because that is mutable. I suppose an append backup snapshot to M-Disc periodically would be boots and suspenders, and frankly the sidecar data is so tiny I could probably append such snapshots to a single disc for years before it all gets used. Although… sidecar data would compress well. Probably simply tgz, then, since it’s always existed, and always will, even if gzip has been superseded by better algorithms.
BTW, I just learned about the b3 hashing algorithm (about which I’m chagrined, because I thought I kept an eye out on the topic of compression and hashing). It’s astonishingly fast - for the verification part, is what I’m suggesting.