can you run something like iperf3 or openspeedtest between the server and client to prove its a network throughput issue?

do you have a network switch you can add to avoid switching through your router (if it is indeed bad?)

Have you ensured you arent unknowingly using wifi at either end?