I was going to suggest there should be a way to run handbrake locally while pointing its under-the-hood functions to handbrake-cli (also, yes, there’s handbrake-cli) hosted on your server, but I found this instead:
medium.com/…/encode-video-with-handbrake-on-serve…. godsspeed, OP