Comment by Houshalter

Perhaps you can weight the flags of users. Like users who have flagged non-spam content in the past don't count, and people who have flagged lots of spam count more.