I guess your OPNSense rule from Edit3 is not working because the source is not your mailu instance, because connections are initiated from the outside and mailu only answers (TCP ACK).

You may get this working if you set the “reply-to” option to the wg gateway on the firewall rule that allows VPS -> wg -> mailu traffic.

However there is a much cleaner solution using the PROXY protocol, which mailu seems to support: mailu.io/master/reverse.html

They are using traefik, but nginx also supports the PROXY protocol.