Eh, that was the whole point. Do not leave moderation to other people or at least make that easy.

It should be relative

Which means that the score of anything would be derived from 1) what you directly set, 2) what another user sets, modified by what you set for that user, 3) what a user sets, modified by what is set for him by another user, which has a value set by you attributed …

One can even make a logic where you see high score for things disliked by people you dislike.

There is some computative difficulty, but nothing big for our times.