To get nginx ingress to use the external clients IP, you can configure the ingress controllers traffic policy. Using the helm chart, I used these values:

controller:
          priorityClassName: cluster-service
          service:
            # this has a bunch of downsides, but allows source-ip based access white/deny listing.
            externalTrafficPolicy: Local

For the ingress IP, I configured metal-lb to receive traffic on a static IP (using IP4AddressPool and L2Advertisement CRDs from metal-lb), which is then used for the port forwarding. I’ve never tested it because I only have a single worker node, but I expect the metal-lb controller will receive traffic to that same static IP if a node goes down.