I dont know much about OpenWRT so dont pay to much attention to me.

I doubt 20ms of extra latency would make much of a noticeable difference. Though its odd its higher now. Does it change, get higher and lower? Are you cable links actually connected at 1000Mbps? Or are some connected at only 100? Can you verify the connection speed?

I have seen IP conflicts with do some weird things depending on hardware. Have a good look to make sure something didn’t get missed with static IPs And Your DHCP pool.