I use a ryzen 3600x and 5600 or 5700xt with 16gb of ram

Solid. My gaming PC runs a 5800X3D, 7900XTX, and 32GB RAM.

My idea was to get a new, smaller case to fit my mitx board and psu in and use the old one with a cpu which supports “all” codes, 32gb ram.

Fair. For what it’s worth, the 3600X will easily support 4K streaming.

The old case has enough space for everything I’ll ever need, but the question is, would it be worth the effort.

That’s highly subjective. Never take on a hobby with the expectation of a return on your investment - you’ll never see it. Do it to learn and further your knowledge.

With transcoding ticked off my issue list, my last remaining point is storage and the uncertainty, whether using usb-c connected, direct attached storage (DAS) systems to set up a fileserver is inherently problematic or not.

My NAS is strictly just a NAS. It’s a 2014 Mac Mini running, Open Media Vault, with a Sabrent DS-SC4B 4-bay hard drive enclosure connected via USB. All 4 drives are in a RAID5 array.

My Plex and Jellyfin instances run within a VM under Proxmox, on an entirely separate machine. It works pretty well for what it is. Though, like you, I plan on moving the whole NAS to a larger case where I can directly connect the hard drives to the motherboard instead of relying on USB.

I don’t understand ZFS

Neither do I, and I don’t have the time nor energy to figure that out. Solidarity!

and docker would yield a ton of chaos if i used it.

Docker is actually pretty sweet once you get the hang of it. I would recommend skipping over docker run commands entirely and going straight to Docker Compose. You write a “compose” file (using YAML) that defines the service, container, volumes, ports, and other (optional) environment variables, find a central place to keep your compose files (separated by folder, because they all tend to be named compose.yaml - it can be anywhere as long as you have the right permissions), point your terminal at whatever compose file you want to run, and tell Docker to fire it up with `docker compose up -d’. And the neat part is that most self-hosted projects will already have an “example” compose file that is easily tweaked to fit your own use case.

There is also a project called “Dockge” (not a typo) that really helps to streamline that whole process with a simple web UI. Made by the same dude who created Uptime Kuma. I run Dockge on everything that runs Docker, including my laptop and gaming PC. They can all be linked together.

Setting up shared network directories in a somewhat polished user interface seems more achievable for me without causing a bottle neck.

For this, I use NFS mounts. OMV makes it pretty easy. Those mounts are then mapped to the appropriate containers inside of my compose.yaml files.

But I had issues when I rebooted VMs / containers with usb pass throughout which took to long to recover. A dedicated NAS would mitigate that issue but would be more costly.

I always advocate for a dedicated NAS, because you can reboot VMs and containers on a separate hypervisor (even the hypervisor itself) willy-nilly without affecting the actual files.

At the moment, I am looking at a terramaster d5 DAS to give my file server a trial…

Looks nice, but I wouldn’t recommend a hardware RAID. If the hardware dies, your data is fucked. With software RAID like mdadm, you can move the array between machines with zero issues as long as the new machine has mdadm installed. It recognizes the array immediately. Really handy.