Peer review is central to how we evaluate science, and most researchers accept that it is inherently subjective. Rejection is part of the process.

What is harder to accept, however, is the lack of transparency in how editorial decisions are made when referee opinions diverge.

Recently, I had three very different experiences in a couple of high-impact journals that I would like to share:

• In one case (a manuscript I am currently reviewing in Advanced Materials), the paper is being evaluated by two referees with an active back-and-forth discussion (only two referees). The editor is engaged and supportive, helping converge toward a decision. The paper is now accepted
• In another case (our previous submission to another high-impact journal), four referees evaluated the manuscript. Three were supportive, one was not. The editor took a clear, constructive stance and guided the process toward a fair outcome.
• In contrast, in a recent submission from our group (in Advanced Materials), the manuscript went through an extended process involving seven referees across multiple rounds. Indulge me the order of refereeing is important. So I’ll detail that.  First reviews came back with referee 1 clearly supportive and saying that this paper should be accepted and giving constructive comments, referee 2 and 3 very appreciative of the work, but not giving a clear decision, referee 4 in a negative tone. We took the opportunity to give a rebuttal, after which referee 2 went missing, referee 3 accepted the paper (referee 1 accepted in the first round itself), referee 4 was now more in disagreement about the length of the paper and several concepts it covers, and less to do with the scientific rigour. At this stage, the editor sought additional opinions, sent it to referee 5, who did not comment (or we didn’t see their response), and referee 6 who has his/her own set of questions. We answered them in another round of rebuttal, this time referee 6 accepted (remember referee 3 and referee 1 have already accepted), referee 4 again a no, and very strangely a referee 7. While we thought referee 6 was the arbitrator, who accepted, still the editor found it compelling to ask another referee, referee 7. Referee 7 said the paper is good, but given too much of back and forth, it should not be published in this journal (not much of scientific critique).

Given so many reviews, there is bound to be opinion differences. More than ever, it is at this point the authors expect supportive editors, especially given that lot of experts have a good opinion on the paper. Or that’s what I thought until the paper is rejected saying there is no unanimity in decision, and with limited clarity on how the differing referee opinions were weighed. This is even after it was contested.

Rejection itself is not the issue. What deserves reflection is the process:

When does adding more referees improve rigor, and when does it introduce noise? In general, sending papers to more and more referees and expecting unanimous decision is designed to be statistically biased against the authors. At some point, adding more referees’ stops being about rigor and starts becoming a lack of editorial conviction in taking a decision.

Questions such as how some papers are getting reviewed by only 2 referees, where as some others, even to begin get sent to 4, then 6, then 7, despite getting overwhelmingly positive reviews?

How should editors weigh detailed positive reports against brief negative ones? Editors should be able to take decisions based on the merit of the reviews, like some of them do. This, however, is not reflected across the board, giving this arbitrariness in how papers are published, and good papers are rejected.

Peer review works best when it is not only rigorous, but also transparent and convergent.

As authors, we invest significant time addressing reviewer feedback in good faith. Publishing in high-impact journals drives us to go beyond our usual limits, interrogating the science in greater depth and presenting the most rigorous, comprehensive version of the work possible, regardless of the final decision. Especially if referee asks questions, we leave no stone unturned in answering them in as much rigour as possible. In return, greater clarity on editorial reasoning, especially in cases of conflicting reviews would go a long way in strengthening trust in the process.

What can the authors do:

Authors feel helpless when such incidents occur. It is because they are helpless. Everything is weighed in favor of editorial decision (esp. after conflicting reviews) not changing. There are few ways to look at this. Take the hit and move on. But this hurts! It also feels insulting. Although the editors always say it’s not the quality of the work in question, one feels that it’s not what they mean, the amount effort put in to disseminate a high-quality work is disrespected, especially for something that undergoes and lengthy multiple referee peer review process.

In case of unfair rejections, the most authors can do (and I started doing this) is to complain, complain and complain. Complain to the handling editor why one thinks the process was unfair, complain to the editor-in-chief, appeal (if time is on your side and students’ side). The editors from some reason become strongly convictional after the appeal, and in most cases, appeals are rejected. But the only thing we can do to improve the way editors handle peer review process is to say we cannot be taken for granted. We will lose, we will lose again, we will lose again and again. But the hope is everything integrates in the background, initiates discussions and may be fires at some point towards more transparency.

Author – Dr. Pavan Nukala