I just finished setting up transcoding for my media library, and I think you’re probably better off using something that’s build for media servers. One really nice feature is adding more processing nodes to make things go faster, like a gaming PC while not playing anything. I don’t think handbrake can do that by itself.

The options I found were Tdarr, FileFlows and Unmanic, but they all use ffmpeg and/or handbrake under the hood, so it kinda comes down to preference. I went with FileFlows because it seemed the most intuitive to me, and it can also process other media like photos, music, audiobooks and ebooks.