I would not enjoy it if a large group of alt-righters suddenly federated with us and became a very vocal presence, even if a large number of their users were often polite, because I am so strongly opposed to those politics.

There is a difference between occasionally annoying people who generally want good things and occasionally annoying people who generally want bad things. One example:

• Alt-right opinions on immigrants in the U.S. range from “let’s immediately deport millions of people and create an even more deadly southern border” to "we should exterminate those [slurs]."
• Hexbear opinions on the same subject range from “we should have an open border, a guaranteed minimum for all U.S. residents, and we should end the imperial meddling that causes so much immigration in the first place” to “yes and the imperial core owes untold sums in reparations to the Global South.”

Maybe you disagree with both, maybe you get annoyed by people talking about both, but one is fundamentally genocidal and the other is fundamentally humanistic.