Welp, turns out I am just an idiot. 1279 and below disabled IPv6, and thus the ::/0 route didn’t get applied either, causing a leak. What’s still odd is the lower download speed that doesn’t happen in another client.
As for the upload, it probably gets a better route through the VPS, giving me a faster speed, and giving me some confusion.

So my first idea with IPv6 was close, but on the other side of the connection.

Anyway, your reply helped me find this issue, as my outtake was to try fully disabling IPv6 (not the first time I tried such “solution”).