Sounds ambitious! Good for you for diving right in.

I’m no expert but I would look at your home internet speeds/capabilities. Most home packages are optimized for download, and if you are self-hosting a streaming server you may a differently provisioned service to support that. If the download to upload ratio is heavily tilted towards download then I would look there first. It sounds like you are able to stream outside your home network, just slowly.

Everything else is waaay beyond me in terms of establishing a secure perimeter and allowing in only desired traffic. I self host a few servers on proxmox, but it’s strictly limited to my home network.