Harvesting IP addresses shouldn’t be a problem, since the firewall shouldn’t allow packets from a peer you haven’t talked to first. But true, if you can be attacked in response by a server you’re connecting to that would be bad.