I'm a PhD and researcher who has worked in various fields, including at a national lab.
I think AI systems like this could greatly help with peer review, especially as a first check before submitting a manuscript to a journal.
That said, this particular system appears to focus on the wrong issues with peer review, in my opinion. I'll ignore the fact that an AI system is not a peer since another person already brought that up [1]. Even if this kind of system was a peer, the system appears to be checking superficial issues and not the deeper issues that many peer reviewer/referrers care about. I'll also ignore any security risks (other posts discuss that too).
A previous advisor of mine said that a good peer review needs to think about one major/deep question when reviewing a manuscript: Does the manuscript present any novel theories, novel experiments, or novel simulations; or does it serve as a useful literature review?
Papers with more novelty are inherently more publishable. This system does not address this major question and focuses on superficial aspects like writing quality, as if peer review is mere distributed editing and not something deeper. It is possible for even a well-written manuscript to lack any novelty, and novelty is what makes it worthy of publication. Moreover, many manuscripts have at best superficial literature reviews that name drop important papers and often mischaracterize their importance.
It takes deep expertise in a subject to see how a work is novel and fits into the larger picture of a given field. This system does nothing to aid in that. Does it help identify what parts of a paper you should emphasize to prove its novelty? That is, does it help you find the "holes" in the field that need patching? Does it help show what parts of your literature review are lacking?
A lot of peer review is kinda performative, but if we are going to create automated systems to help with peer review, I would like them to focus on the most important task of peer review: assessing the novelty of the work.
(I will note that I have not tried out this particular system myself. I am basing my comments here on the documentation I looked at on GitHub and the information in this thread.)
[1] https://news.ycombinator.com/item?id=44144672
I think AI systems like this could greatly help with peer review, especially as a first check before submitting a manuscript to a journal.
That said, this particular system appears to focus on the wrong issues with peer review, in my opinion. I'll ignore the fact that an AI system is not a peer since another person already brought that up [1]. Even if this kind of system was a peer, the system appears to be checking superficial issues and not the deeper issues that many peer reviewer/referrers care about. I'll also ignore any security risks (other posts discuss that too).
A previous advisor of mine said that a good peer review needs to think about one major/deep question when reviewing a manuscript: Does the manuscript present any novel theories, novel experiments, or novel simulations; or does it serve as a useful literature review?
Papers with more novelty are inherently more publishable. This system does not address this major question and focuses on superficial aspects like writing quality, as if peer review is mere distributed editing and not something deeper. It is possible for even a well-written manuscript to lack any novelty, and novelty is what makes it worthy of publication. Moreover, many manuscripts have at best superficial literature reviews that name drop important papers and often mischaracterize their importance.
It takes deep expertise in a subject to see how a work is novel and fits into the larger picture of a given field. This system does nothing to aid in that. Does it help identify what parts of a paper you should emphasize to prove its novelty? That is, does it help you find the "holes" in the field that need patching? Does it help show what parts of your literature review are lacking?
A lot of peer review is kinda performative, but if we are going to create automated systems to help with peer review, I would like them to focus on the most important task of peer review: assessing the novelty of the work.
(I will note that I have not tried out this particular system myself. I am basing my comments here on the documentation I looked at on GitHub and the information in this thread.)
[1] https://news.ycombinator.com/item?id=44144672