Otherwise, it’ll just fill up with all sorts of crap from communities with no downvoting rules, including edgy borderline racist stuff that’s not quite bad enough to get banned.

I may be wrong, but admins will be able to configure what communities should be visible in the public view. So your instance would not show on their frontpage things that are not representative of the instance

For users themselves who are browsing by /all and feel justified in downvoting because they don’t like what they see, it’s a different story. If a community is (in their view) problematic, they can simply block it. Downvoting has no place in their curation.