Hmm... well Peerage of Science was an overlay on top of the journals. It was basically a system the journals would pay for to mediate their own review. Maybe similar tech-wise, but very different goals.
> Speaking for myself, I don't trawl through preprint servers looking for papers to review.
> As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time.
Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus.
Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system.
> How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now.
See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action.
> It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time.
> It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses.
That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation.
> Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible?
> What's the negative consequence should someone not provide a response?
It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes?
Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning?
> Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs?
We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print".
> Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one?
- PubPeer is just a comment overlay, can't read the papers (as I understand it).
- PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ).
- Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues.
- Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access.
None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with.
But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!
> Speaking for myself, I don't trawl through preprint servers looking for papers to review.
> As it is, editors have a hard time getting people to review, and even after agreeing to review, to get the review back in reasonable time.
Could we change that culture? The hope would be that we could build a culture of people helping each other review, for the simple reason that they want that help themselves when it comes time to for them to publish. With the reputation being more of a bonus.
Keep in mind, review here is purely editorial feedback. It is not refereeing - that happens post publish through the voting system.
> How does it handle peer review rings, where authors collude with reviewers to get bogus positive reviews? This is already an issue now.
See above - review is not refereeing. It could be an issue in refereeing in the votes for sure though. The hope is that the overall weight of the community would override those rings. And in the long term we might even be able to identify them through analysis of the database. If we see the same group of people always up voting each other's down voted papers, we can flag that and take corrective action.
> It offers reviewers an incentive to write a review that the paper author likes. I expect if reviewer writes "this paper duplicates work by Penzias & Wilson (1965) and offers nothing new - reject" then the author will reject it. While "Fix the typo on page 4, otherwise, go ahead and publish" will be accepted - even if the paper really is a waste of time.
> It's a lot easier to get surface-level proofreading responses than substantive domain-specific responses.
That may turn out to be true, but the thought is that the post-publish refereeing system will help incentivize people to take critical feedback seriously at the review step. After all, your goal is to publish something that will be well received. If you get good constructive critical feedback and ignore it, you risk being downvoted to oblivion and losing reputation.
> Why aren't up voters strongly encouraged to post a response explaining their upvote? Why should upvotes not be equally visible?
> What's the negative consequence should someone not provide a response?
It's been asked before and it's definitely on the table. It's something I'd love more feedback on - should responses be required for all votes? Just down votes? Or (as I'm defaulting to) should we follow the StackExchange model of just encouraging responses on down votes?
Seems maybe you would argue in favor of requiring responses on all votes? What's the reasoning?
> Any reason to not develop this as an overlay journal on top of existing pre-print servers? Thus reducing a lot of the costs?
We don't want it to be an overlay, because we want to ultimately replace the system. But it could make a lot of sense to work with the existing pre-print servers. I see no reason why we couldn't start as a pre-print and gradually evolve towards being the final publish, as long as we're clear about the end goal so that we don't just become "yet another pre-print".
> Why would anyone use this system given PubPeer, and PeerJ, and eLife, and overlay journals, and Publons, and all of the other attempts at breaking the current system, why would someone choose this one?
- PubPeer is just a comment overlay, can't read the papers (as I understand it).
- PeerJ is still using the manual model, hence overhead and fees to publish ($1200) rather than diamond open access. This creates equity in access to publishing issues ( a lot of people cannot afford those fees ).
- Elife, similarly, charged $3000 per article. These fees create massive equity in access to publishing issues.
- Publons I wasn't previously familiar with, but it seems to be about just highlighting review work done for the journals, and not anything to do with open access.
None of the attempts to break the current system (that I've found) seem to really be able to replace the journals by doing (almost) everything the journals do. At least, not with out just being a journal and facing the overhead problems. The one thing this project doesn't do that the journals do is curate reviews/refereeing, but the idea is that crowdsourcing could actually work better in the long run - though it's definitely going to bring its own set of problems that will need to be dealt with.
But thanks! I hadn't heard of Peerage of Science, and Publons sounds vaguely familiar but I wasn't really familiar with it. I'm constantly learning more about what other efforts have come before and currently exist (there are a lot of them). And thinking through all the questions and counter points is always very helpful!