Comment on Data organization

Scipitie@lemmy.dbzer0.com ⁨2⁩ ⁨days⁩ ago

Hey, Welcome! You’re having a wonderful and sometimes exhausting journey in front of you :)

I’ll just brain dump based on your questions and my associations. Hope something useful is in between!

First the basic setup options because they tie into how to handle your date flow:

basic Most popular I think is docker compose: here I suggest splitting it into one compose file per service though with one file holding your port config. This prevents you yourself getting confused by your port mappings :)

Second in line is a proxmox setup - similar vein and I lack the hands-on experience to talk about the difference.

Then there’s the “everything native” approach where you don’t rely on containers but manage it yourself or via a dedicated OS that makes life easier (after the learning curve) like nixos.

DATA

All this foundational stuff is important because it changes your approach. In general: don’t fear data duplication. Duplicate it until you learn where you want your data to life and only then define your flow.

Specific example: after I got used to paperless I don’t look into my opencloud anymore, at all. I still duplicate them there but as distributed backup, not for consumption.

If a dataset has a clear place ten it’s easy. If not then your options are different depending on your setup: For the *arr stack the official recommendation is to use one shared folder and mount that into each part for example. I personally don’t like that and have hard links for everything - that’s basically a pointer to the file that looks like the file itself everywhere. As long as one pointer exists the file still stays on your drive but when the last pointer is gone, the file is effectively deleted. On Linux, you can think of every file this way but by default only one pointer exists (which often people test as synonymous to “the file”. Drawdown: this only works really well if you manually keep either track of which tool links where or you don’t containerize everything.

Again a specific example: My downloaded torrents never get moved - instead hard links are created into whichever path and naming scheme I defined for each consumer - this way, out of murdrrbot_07.mp3 a new author/series/booktitle.mp3 was created, both pointing to the same data and seeing it as a proper file.

But then there is one more thing: I suggest you split your thinking into data consumption and manipulation - because for the first, data duplication doesn’t matter. Especially for documents you’re talking about a ridiculous small amount of disk space and if it’s only reading/watching/hearing you as manager have no problem that data might exist multiple times.

If you want to keep it clean by design then you’re leaving the starter mode self holster - welcome to system design and infrastructure architecture! Here your approach could be to define lifecycles for each data type that you have. What a “data type” is in this context btw is a user term, NOT the underlying tech stack. You need to understand and document how an invoice should be treated and consumed by you differently than an invitation or a informal letter. Only then do you map file types, incoming channels, transformation steps, etc etc.

In my opinion: huge overkill to this upfront.

In short: spin everything up, observe how you use it and only then decide where things need to stay unique and cleaned up. Don’t break your head over something that’s actually quite easy to repair!

original
Sort:hotnewtop