If votes became truly public

There’s no ‘if’, they already are.

what would stop a malicious user from automating crawling the fediverse to get a list of every up and down vote a targeted user has ever made?

The same thing currently stopping them: nothing but time and effort.

Admins can currently do this, I assume given enough time and intent? Yuck.

No, anybody can currently do this.

That’s the issue with decentralisation. The info is out there. It’s that or trust a megacorp with it.