Kinda unrelated to OOP, but out of curiosity, what does your storage setup look like? Do you keep stuff reasonably backed up with that much data?
Comment on How big is your media library?
SirMaple__@lemmy.ca 1 week ago
1,152 - Movies 552 - Shows 37, 062 - Episodes 491 - Albums 6,558 - Songs 362 - Music Videos 14 - Concert Films
Files are a mix of 1080p and 4K. 264 and 265. Standard and REMUX.
Total space used is currently 149.90TiB
marighost@piefed.social 1 week ago
SirMaple__@lemmy.ca 1 week ago
Ah yes. My storage system is 2 x Supermicro CSE-846 cases. Only one as a CPU and motherboard, the other is acting as a plain Jane JBOD.
Hard drives I have 21 x 8TB 7200RPM mix of Seagate and Western Digital and 4 x 16TB 7200RPM from Seagate. I use mergerfs and snapraid. Mergerfs presents all the 21 8TB drives as one mount point. Snapraid uses the 4 16TB drives to provide 4 parity drives. Note that snapraid is not live and the parity is only updated after running a “snapraid sync” which I run nightly.
I only backup my songs and music videos. The rest is easy to get again. I have a script that generates a list of every single file I have each night. So if the day comes it would take too long to get back to where I was. The other reason I use mergerfs is if 1 drive dies, I only lose the files on that one drive and not the entire array. The truely important stuff such as tax documents, mortgage details, family pictures, will & estate documents are all backedup nice a safe using Proxmox PBS. The PBS datastore is synced to 2 remote locations as well as to external drives that I keep offline and rotate.
Policeshootout@lemmy.ca 1 week ago
Nice write-up. I thought I had a large library (24TB) and my off site backup is starting to get full. I backup everything though but I have long debated on if there’s a point of keeping movies and TV since they’ll likely always be available. Anyway, I never thought of generating a list of files and eliminating the stuff that’s not particularly important. Good idea.
SirMaple__@lemmy.ca 1 week ago
I used to back everything up before I broke the 50TB mark. Just can’t justify it now. I even looked at LTO drives for backing up the multimedia but they’re still to expensive for the higher capacity drives. And then you need tapes…
All the truely good content will always be out there somewhere on the net.
The script I use to generate the file lists is very very basic. Nothing special no formatting the lists or anything since it’s just for that oh balls, everything is gone scenario.
ls -alR /mnt/volume1/media > /mnt/volume2/backups/file_lists/media.txt
ls -alR /mnt/snapraid/data* > /mnt/volume2/backups/file_lists/snapraid.txt
Those text files are also part of the files backed up with PBS so I can always go back and restore previous versions of them. You may ask why I generate the list twice? The first is just everything inside the media folder on the volume1 mount point. The second let’s me see what files are on each individual drive so if only 1 drive dies I can just grep the text file and output to another text file and show me what is on that 1 drive.
marighost@piefed.social 1 week ago
Thanks for the detailed write up!
curbstickle@lemmy.dbzer0.com 1 week ago
About the same here, minus the music videos (only a few dozen there for the kids), plus a fitness library, so I’d say it evens out to roughly equal.
Mostly HEVC but I still have some h.264 floating around that I have no interest in reencoding.
No AV1 at all until I get a new Intel GPU or newer Intel CPU to handle transcoding it nicely.
SpookyMulder@lemmy.4d2.org 1 week ago
There’s some relatively inexpensive NVIDIA cards now with AV1 hardware encoding. I’m on my third round of re-encoding my whole library (HEVC, then VP9, now AV1). For 1080p NTSC, I get about 13x speeds on NVENC AV1, whereas with VP9 I was CPU-bound at around 4x. Definitely worth the upgrade, in case you’re on the fence.
TheYang@lemmy.world 1 week ago
You do realize that you lose quality with wach encode, right?
It’s not AS bad when bitrates are high, but it’s still there.
SpookyMulder@lemmy.4d2.org 1 week ago
True.
When I migrated off of Jellyfin, I re-encoded everything up to that point directly from the Blu-ray rips wherever possible. Because I’d already started culling those for space, I did end up just doing another pass on the first round of encoding for a portion of the library. There’s some noticable degradation on those, and I’ll want to re-rip those at some point.
Fortunately, I’ve got my process pretty dialed in for ripping and I actually enjoy it, so if I ever have a quality issue, it’s not a huge ordeal to re-rip and encode.
Onomatopoeia@lemmy.cafe 1 week ago
So what card are you using?
SpookyMulder@lemmy.4d2.org 1 week ago
I went with a GeForce RTX 4060. Cost was about $300.
curbstickle@lemmy.dbzer0.com 1 week ago
Yeah I just dont have a need with no devices to handle it natively, while the rest of my library can be. Building a new htpc media player for the living room next, new server after that.
New because I’m using a lenovo tiny as the server, which means either I build a new box completely, or I find the right used workststion tiny/mini/micro that can handle av1. Complete build will do a lot more (well, the t/m/m does too, but not to the extent my big box builds are set up for).
SpookyMulder@lemmy.4d2.org 1 week ago
Ahh, I like how you split Concert Films and Music Videos. I’ve been pigeon-holing my Short Films, Mini-Series, and TV Movies into just the two categories: Shows and Movies. Makes way more sense having separate categories.