I need help troubleshooting slow wireguard tunnel (VPS to home network)

Submitted ⁨⁨3⁩ ⁨days⁩ ago⁩ by ⁨stavefajl@feddit.dk⁩ to ⁨selfhosted@lemmy.world⁩

Hello I hope one of you can help point me in the right direction.

I have a VPS with a static IP and a wireguard tunnel from VPS to home network (no bridging in the router, just point-to-point with specific devices).

I found an abysmal connection speed with bandwidth on the order of 50-100 kbps tested via iperf. Connection between the same devices outside the wireguard tunnel is 10-20 mbps, which is 100-400 times slower, which I don’t understand since wireguard usually has very little overhead.

I have tried different MTU settings on both VPS and devices on my home network (both cabled and via wi-fi) in the range from 1360 to 1460, and above speeds are the best I have reached with MTU 1420 and 1440. I have tried both with and without iptables rules setting the mss correspondingly.

The above speeds are acceptable for incremental backups and document synchronization, but completely unsuitable for media streaming.

Where would I start diagnosing the bottleneck?

Thanks in advance.

source

Comments

Sort:hotnew top

tychosmoose@lemmy.world ⁨3⁩ ⁨days⁩ ago
Are you specifying bandwidth (-b) on the iperf UDP test? It defaults to 1M if I recall correctly, which would explain the result.

If not, try -b 10M or -b 0 for unlimited (the behavior used for TCP).

source
- stavefajl@feddit.dk ⁨2⁩ ⁨days⁩ ago
  Thanks! I had not read the manpage close enough, I guess. When specifying the bandwidth, I can saturate the connection using UDP to around 15 Mbits/s. (That this speed is much much lower than the 300-500 Mbits my connection and the VPS is capable of is a problem for a different time, I think).
  
  What I also realized is, that I had not put the iperf server in UDP-mode, so my results reported in another comment are wrong. I read the results from the client, but the server did not respond. When running the iperf server in UDP-mode, I get 15 Mbits/s outside the wireguard tunnel and 180 Kbits/s inside the tunnel. With TCP-mode I get 10-15 Mbits/s outside the thunnel and 250 Kbits/s inside the tunnel.
  
  source
  - tychosmoose@lemmy.world ⁨2⁩ ⁨days⁩ ago
    Just throwing out more ideas:
    
    Is there a CPU spike on the VPS?
    
    Anything weird about Wireguard on either end? Using kernel mode WG everywhere and not a user mode version, right?
    
    As a test I would be inclined to try a very small mtu to see if it makes a difference. 1280 is a failsafe that I use when on unknown networks and trying to wg out.
    
    Maybe try with a smaller packet size, like 1KB which I think is -l 1K
    
    source
    -> View More Comments
Brkdncr@lemmy.world ⁨3⁩ ⁨days⁩ ago
What iperf speeds are you getting with UDP?

On windows you can use the test-connection -MTUSize powershell command to figure out the correct mtu.

If your udp speeds are normal but your tcp speeds are not, it could be fragmentation, or out of order packets, or high latency.

source
- stavefajl@feddit.dk ⁨3⁩ ⁨days⁩ ago
  I’m on linux on all devices. I tested packet fragmentation using:
  ping -M do -s [packet size] [ip] No fragmentation using packet size 1472 and below (before wireguard overhead).
  
  source
- stavefajl@feddit.dk ⁨3⁩ ⁨days⁩ ago
  I just tested again:
  
  no wireguard, tcp: 9.75 Mbits/s
  
  no wireguard, udp: 1.05 Mbits/s
  
  wireguard, tcp: 248 Kbits/s
  
  wireguard, udp: 1.05 Mbits/s
  
  So, I get the “full” udp speed, but I get some errors / warnings about ‘connection refused’ and ‘did not receive ack’. Obviously not correctly configured, when it is 10 times slower than tcp.
  
  source
  - Brkdncr@lemmy.world ⁨3⁩ ⁨days⁩ ago
    What’s your mtu inside the vpn?
    
    source
    -> View More Comments
stavefajl@feddit.dk ⁨3⁩ ⁨days⁩ ago
I am currently running mtr from multiple devices:

in wireguard tunnel: single hop 10.1.1.1, loss% 0,1, avg ping 27.4 ms

outside wg between same devices:

ISP supplied modem/router 55% loss, avg ping 1.6 ms

multiple hops without loss, avg ping 16-20 ms

random intermediary 30 % loss, avg ping 20 ms

endpoint, 5% loss, avg ping 25 ms

It looks like across the board my ISP modem / router is dropping 50-80 % of packets, and that packet loss is ramping up from 4% to 80 percent after a few minutes of running mtr.
It also looks like my VPS endpoint climbs to 20% packet loss over time (5-15 minutes of testing).

Can I use this information to probe further into the devices I have access to (ISP modem and VPS)?
source
- Brkdncr@lemmy.world ⁨3⁩ ⁨days⁩ ago
  Some devices will drop icmp packets. Does iperf show the same amount of loss?
  
  Can you try a different internet connection such as mobile hotspot just to see if you have similar results? That could help identify it as a vps issue.
  
  source
  - stavefajl@feddit.dk ⁨2⁩ ⁨days⁩ ago
    I had to borrow a phone to set up a mobile hotspot.
    It has the same speeds inside the wireguard tunnel as when I tested from my wired connection (250 kbps TCP, 170 kbps UDP).
    The loss reported by iperf is dependent on the bandwidth that i test with. But as I increase bandwidth from the client the loss grows towards 100%.
    
    I tried testing in reverse (sending from VPS to devices on different networks) with surprising results:
    
    TCP, wireguard: 5-10 mbps
    
    UDP, wireguard: 50 mbps
    
    TCP, no wireguard: 45 mbps
    
    UDP, no wireguard: 250 mbps (saturates download speed on client when compared to speedtest.net)
    
    source