The best part of the fediverse is that anyone can run their own server. The downside of this is that anyone can easily create hordes of fake accounts, as I will now demonstrate.
Fighting fake accounts is hard and most implementations do not currently have an effective way of filtering out fake accounts. I’m sure that the developers will step in if this becomes a bigger problem. Until then, remember that votes are just a number.
That sounds like a solution that basic administration/moderation is meant to solve. You need to feel how bad brigading is to understand the need to not be able to downvote/see downvotes. Other instances can see the votes, which keeps the system honest and removes the need for individual users to have transparency.