Each one of them did nothing but downvote everything, and never contributed at all to the community.

Downvotes are a contribution, they are just the kind of contribution you don’t like. based on this, I don’t think you area good fit for modding; you should probably look to pass your role on to someone who can moderate responsibly.