It would be annoying to do that for everyone who uses my wifi.

I found that “Conversations” (XMPP) was blocking a connections even when the permission was enabled. I suspect Android no longer allows connections on port 5xxx on a local network. It works fine when the IP is external.