We have done a similar thing and build a tool to just do this in house. We have a team of human moderators who look at each review and do moderation.
Took a few weeks for a dev to build a hacky solution. Would have liked to see it then, may be we could have given you a try.
Curious, How is this similar to GitHub ?