You’re not using a CPU that most distributions support for hardware transcoding. You either need to use an Intel CPU with QuickSync or stick a discrete nvidia card in the box. The Intel route is often easiest here, and I say this as a die hard AMD fan.

Are you intent on building your own box?

I’m only asking because TerraMaster does the F6-424 (or F4-424) series which has 6 bays (or 4), a decent CPU (1235U) with hardware transcoding support, space for two 4x4 NVME m.2 SSDs, which runs silently and will just work as an appliance, even though it is a full PC. You can then install unraid or truenas on it, or heck bareback Linux and do it yourself. There are decent alternatives to putting something together yourself.