I’ll have to look at my fail2ban logs and see if I’m having similar issues.

It should be possible to mod your jail to whitelist an IP range on your local Network.

I’m doing that on one of my jails.