Would distributing all reviewers’ reports for a specific paper amongst every referee before deciding whether to accept or reject a manuscript make peer review fairer and quicker? This idea — called “cross-referee commenting” — is being implemented by the journal Development, as part of its attempt to improve the peer-review process. Katherine Brown, executive editor of Development from Cambridge, UK, who co-authored a recent editorial about the phenomenon, spoke to us about the move.
Retraction Watch: Many journals share the reviews of a particular paper with those who’ve reviewed it. What is cross-referee commenting in peer review and how is it different from current reviewing processes?
Katherine Brown: Indeed, sharing referee reports after a decision has been made is common practice across many journals, and can provide referees with valuable feedback on how their assessment of a paper compared to that of the other referees. What’s different with cross-referee commenting is that we are sharing the reports before a decision has been made, and asking for feedback — does each referee agree with the others’ reports, and if not, why not? The idea behind this is to help the editor decide whether or not to invite a revision and, where the decision is positive, to help direct the authors as to the most important revisions required for eventual acceptance of the paper.
RW: What made you decide to introduce cross-referee commenting?
KB: In late 2015, Development and the other Company of Biologists journals surveyed their authors and readers about how we could improve the peer-review process. The survey revealed that what matters most to authors is how quickly we can move a paper from submission to final decision, and this is something we are constantly trying to improve. But, speed aside, the most popular potential innovation was to trial some kind of collaborative peer-review process. Such models are already in place at a number of other journals (see e.g. Pulverer, 2010 and Schekman et al., 2013). In surveying possible options, we wanted to ensure that we did not significantly compromise speed to decision, or overburden referees who already put a lot of time and work into reviewing papers. We therefore decided that a cross-referee commenting model where we request that any comments be returned within 48 hours was the most appropriate way forwards for us. This is now being implemented on both Development and our sister journal, the Journal of Cell Science.
RW: Ultimately, your project’s aim is to “resolve differences between referees, identify unnecessary or unreasonable requests, or — conversely — highlight valid concerns raised by one referee but overlooked by the others.” How common are these problems? And how are they addressed by cross-referee commenting?
KB: In most cases, referees are broadly in agreement and a decision on a paper is straightforward. For these papers, cross-referee commenting is probably not necessary, although we still feel that it might help the editor to clarify the most important criticisms or suggestions. However, there are clearly some papers where referees disagree — either on how much of an advance the paper makes, or on how solid it is. Here, further feedback can be hugely valuable, and in fact we have already been conducting informal cross-referee commenting in such cases for several years. We all know that we occasionally miss an important point in an argument, or have an entrenched view that perhaps is out-of-date — these are the kinds of things we’re trying to address.
With cross-referee commenting, referees can see where their opinion is out of step with their peers — in some instances, this might lead them to acknowledge that they were too positive or negative in their initial assessment; in others, they might be able to explain more clearly why they hold a particular point of view. In both cases, this provides valuable additional information to the editor, whose job it is to weigh up such differences of opinion and decide how to proceed. It’s hard to put numbers on how often cross-referee commenting might lead an editor to change his or her decision — I think this will be pretty rare — but I think it will quite frequently help the editor to provide better advice to the authors on how to revise their paper. So although we think cross-referee commenting will only have a significant impact in a minority of cases, we felt it would be fairest to introduce it across all papers.
RW: You said in your editorial that peer-review reports often read like “shopping lists.” What do you mean by this?
KB: I think there can be a tendency for referees to ask for too much — it’s always possible to think of additional experiments that authors could do. But in many cases these are experiments that would extend the study in various directions, rather than ensure that the current conclusions are sufficiently solid. So a review can sometimes ask the authors to do things that not only will take them many months, but also would likely form the basis of a whole new paper (or papers!) — rather than being a core requirement for this particular paper. In 2015, we changed our referee report form to try and avoid this. What we want referees to focus on is how well the data in a paper support the conclusions, rather than how the authors can make those conclusions more ‘interesting’. We think of this as making sure the referee limits his or her comments to the ‘necessary’ revisions rather than the ‘nice-to-have’s’. So in the shopping list analogy, the referee report should contain the milk, bread and vegetables (the things you need), but avoid the chocolate and wine (the luxuries)!
RW: How have authors and reviewers reacted to this announcement?
KB: The response to our announcement has been hugely positive, from both the Development and the Journal of Cell Science communities. We think this is because we’re making this change in response to their views — as a direct response to our survey. We’ll be rolling out the new process in the next week or two and then we will get to really see it in action. But we know from other journals that it works well and is generally popular.
RW: Will the new reviewing process at your journal affect referees’ option of remaining anonymous (by not signing their reports)? If so, how?
KB: This was something we thought hard about, and decided to retain anonymity: Unless referees choose to reveal their identities, they will remain anonymous to both the other referees and the authors. At this point, we don’t believe that open peer review would help to improve the process, and we suspect that some referees might decline to review if their identities would be known to other parties. Anecdotally, we’ve been told by referees that open discussions can be uncomfortable, and our survey reinforced the lack of enthusiasm for open peer review in our communities — this came out as one of the least popular potential innovations.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
If its any good at all, better that each comes to their own conclusion independently and reduces it to paper. If they want to share content in a moderated discussion it might work
then since they have already have a considered conclusion at stake and are less likely to
be swayed by the power member of the peer group in the presence of an independent moderator.
Agreed – I think it’s important that referees should form and record their independent opinion of a paper before any discussion happens – otherwise there is a risk of ‘he who shouts loudest wins’. The way we’re implementing this is designed to minimise the risk of this, and I’m confident that our editors can act as strong arbiters of any potential disagreement.
The principle is good, and would indeed allow for consensus regarding merit and problems, or limitations. However, one key element is missing. The public does not know what is shared or agreed/negotiated by and between peers. Although the process is likely to be quite rigorous and strict at Development, a hidden negotiation between and among peers in more suspect journals might be a source of bias (for or against a set of authors). What would really make this cross-peer discussion valuable is if, after an acceptance decision, the peer reports at all stages (i.e., before and after cross-peer interaction) could be published as open access reports (i.e., maintain the peer review closed, if that is what the preferred model is), but then make the reports open finally.
Publishing referee reports (and other relevant correspondence) alongside published papers came out pretty low down the list of priorities in our community survey, so it’s not something we’re planning right now, but we haven’t ruled it out for the future. Transparency is certainly important to demonstrate that peer review is working well, but I don’t think it actually makes much difference to the process itself – which is, after all, what’s most important!
It would be more interesting if a paper could be submitted online together with original data and any scientists working in the field could submit their opinions during a certain period of time and anyone could have the possibility to read the comments. After this period of time a selected group of scientists would analyze the different opinions and get to a conclusion about the publication of the paper
Does she honestly believe that one referee would have the balls to say to another, “What you’re asking of the authors is unreasonable”? These are all researchers in the same sub-field, who have to interact with and referee each other’s papers.
Yes absolutely – I’ve seen it happen, particularly when an editor specifically asks a referee if they think other requests are reasonable. But this is why I think anonymity is valuable, and why comments are filtered via the editor rather than open debate happening between referees. There are journals (eLife) that engage in fully open discussion among referees, and it can work very well, but I fear it can also put pressure on referees in the way you describe…
The reviewers should keep their independency. It might be better that all reviews should be made open, i.e., including the name of the reviewer, i.e. not anonymous. The reviewers can then decide by themselves, whether they want to contact each other or not. And the authors do know who did the review.
In my field almost all papers that are not desk-rejected go through at least two rounds of review. Some journals now circulate all first-round reviews, after they have been submitted and together with the editor’s first round decision, to all referees. This stops short of requesting cross-comments, and maintains referee independence in the first round, while letting me as a referee see what the concerns of other referees have been. This can have the effect of moderating unreasonable requests (I can see that I am pulling in one direction, another referee in another, and that the combined demand is unreasonable – this in keeping with what Katherine Brown is saying about refereeing the paper you have, not the ‘more interesting’ one that it suggests to you); and, since different referees notice different problems, it gives me a better sense of the overall quality of the paper (and, in the end, teaches me a lot, which makes refereeing slightly more worthwhile).
Short answer… NO!
The NIH specifically moved to stop this around a decade ago. It used to be that study section reviewers would find out who the other reviewers were on their assinged proposals, shortly ahead of the in-person meeting. That way, they could iron out any differences ahead of the meeting, thus keeping the meeting shorter by minimizing lengthy discussions about disagreements.
Nowadays, you have to wait until the meeting to see who the other reviewers are. This is much better in my opinion. Under the old system there was the possibility for reviewers to bully each other into submission. Especially as a junior member on study section, I found it very difficult to disagree with senior persons in the field, via email or phone ahead of time. You can imagine the kind of inappropriate things that may be said off-line, out of the gaze of the SRO and the other members of the section. Throw in issues of race and sex on top of seniority, and there’s potential for a lot of off-line bullying, dressed up in language such as “toe the line junior.” Yes, it gets the scores in line, but the “winner” is almost always on old white dude. By delaying such confrontations until the actual meeting itself, the chance for such shenanigans is greatly reduced – people are more likely to be respectful of a junior scientist’s opinion, and less likely to behave like privileged ****s, when they’re in a room full of people.
So by all means have reviewers talk to each other, but it HAS to be in a monitored forum, not off-line behind the scenes. The last thing we need is to have senior reviewers ganging up on junior reviewer #3 who has “found a fatal flaw in the paper, but doesn’t really know what she’s talking about, and clearly doesn’t appreciate the impact of this work from our esteemed colleague”
I have sympathy with these concerns, and this is one of the reasons why the model we’re adopting is NOT an open discussion between referees. The idea of fully open peer review is nice in theory, but I do worry about the potential for it causing grudges to develop, and perhaps even for bullying behaviour. And I don’t think this just applies between authors and referees, but also among a group of referees. Because our model restricts communication to being between editor and referee (and is all recorded in our online submission system), it should provide a ‘safe’ forum for referees to put their opinion forwards without fear of repercussion.
This is not new and in fact is already done at journals like eLife (where I have served as a reviewer) and eNeuro (where I am one of the reviewing editors). The process works well in all the cases I have seen so far.
Firstly the peer review has become faulty, because most of the journals are asking teh suthor to suggest the reviewer, and naturally one will suggest reviewer which will favour the author, my many ways, including field being same.
In case if any reviewer has conflict of interest with the author then this system will fail.
In ideal condition, cross talk by reviewer will be beneficial to improve the quality of the paper. Surely.
No, each to its own knowledge and experience and ganging up on a contributor-No field Mafia
Some journal ask the name of potential reviewer at time of submission of manuscript.This must be stopped. Sharing reviewer comments among other reviewers is helpful and a Good idea.