Wireguard is not meant to bypass detection of any kind of Deep Packet Inspection technology. It's also stated by the project website, under the Known Limitations sections.
https://www.wireguard.com/known-limitations/
There are many characteristics (except the handshake) which can make it stand out as a P2P or VPN(-ish) traffic, and then get blocked.
mfat@lemdro.id 1 year ago
I haven’t tried this. Will do. When wireguard still worked i used this script to set it up.
Max_P@lemmy.max-p.me 1 year ago
Quick glance at the script shows it does randomize the port (but in the upper end of the space), but does not do preshared keys.
The PSK can hide the protocol better by not looking like a key handshake. Although I’m not sure ISPs have quite reached that level yet.
If you’re in China, there’s been some recent research showing that they block things that looks encrypted based on the entropy of the bits in a packet. I couldn’t find the bypass software anymore but that might help you as well.
If you’re not in China something like the XRay/v2ray proxy or shadowsocks might help as well.
Although honestly, port forwarding with a firewall only allowing your VPS IP should honestly be fairly sufficient security-wise. You could also try inverting the WireGuard connection and have the VPS connect to your home through port forwarding, traffic analysis is usually on egress not ingress.