Private voting across the fediverse, where plenty of users have multiple accounts spread out across the fediverse on different instances would invite a ton of mass-downvoting as it could be done with relative ease.

If it is proper coordinated or autmated mass-downvoting campaign that should be a job for the admins.

And it’s my legitimate use as a moderator to determine that someone who does that is not part of any community I am building, and is actively vandalous towards it.

Yes, that’s how it currently works. I’m arguing it shouldn’t work like this for votes.