There is no such thing as a “vote” in ActivityPub/ActivityStreams. This idea of “up/down votes” is just an abstraction of a message saying “Actor A liked B”, where B is an Post/Comment (and a post/comment itself just being an abstraction of ActivityStreams objects).

That is to say: there is no way to selectively hide the content a message. If you want federation to work and you want people outside your own server to see your posts, then the server needs to broadcast the messages to anyone listening.

Tools like lemvotes are just exposing this information. There is no point in trying to censor the tool, because this information is available publicly, and any motivated person will be able to track this information.

If you are concerned about what people think of your “likes” and “dislikes”, then do not use a public social media service and only communicate with provably secure communication tools.