Would distributing all reviewers’ reports for a specific paper amongst every referee before deciding whether to accept or reject a manuscript make peer review fairer and quicker? This idea — called “cross-referee commenting” — is being implemented by the journal Development, as part of its attempt to improve the peer-review process. Katherine Brown, executive editor of Development from Cambridge, UK, who co-authored a recent editorial about the phenomenon, spoke to us about the move.
Retraction Watch: Many journals share the reviews of a particular paper with those who’ve reviewed it. What is cross-referee commenting in peer review and how is it different from current reviewing processes?
Katherine Brown: Indeed, sharing referee reports after a decision has been made is common practice across many journals, and can provide referees with valuable feedback on how their assessment of a paper compared to that of the other referees. What’s different with cross-referee commenting is that we are sharing the reports before a decision has been made, and asking for feedback — does each referee agree with the others’ reports, and if not, why not? The idea behind this is to help the editor decide whether or not to invite a revision and, where the decision is positive, to help direct the authors as to the most important revisions required for eventual acceptance of the paper.
RW: What made you decide to introduce cross-referee commenting?
KB: In late 2015, Development and the other Company of Biologists journals surveyed their authors and readers about how we could improve the peer-review process. The survey revealed that what matters most to authors is how quickly we can move a paper from submission to final decision, and this is something we are constantly trying to improve. But, speed aside, the most popular potential innovation was to trial some kind of collaborative peer-review process. Such models are already in place at a number of other journals (see e.g. Pulverer, 2010 and Schekman et al., 2013). In surveying possible options, we wanted to ensure that we did not significantly compromise speed to decision, or overburden referees who already put a lot of time and work into reviewing papers. We therefore decided that a cross-referee commenting model where we request that any comments be returned within 48 hours was the most appropriate way forwards for us. This is now being implemented on both Development and our sister journal, the Journal of Cell Science.
RW: Ultimately, your project’s aim is to “resolve differences between referees, identify unnecessary or unreasonable requests, or — conversely — highlight valid concerns raised by one referee but overlooked by the others.” How common are these problems? And how are they addressed by cross-referee commenting?
KB: In most cases, referees are broadly in agreement and a decision on a paper is straightforward. For these papers, cross-referee commenting is probably not necessary, although we still feel that it might help the editor to clarify the most important criticisms or suggestions. However, there are clearly some papers where referees disagree — either on how much of an advance the paper makes, or on how solid it is. Here, further feedback can be hugely valuable, and in fact we have already been conducting informal cross-referee commenting in such cases for several years. We all know that we occasionally miss an important point in an argument, or have an entrenched view that perhaps is out-of-date — these are the kinds of things we’re trying to address.
With cross-referee commenting, referees can see where their opinion is out of step with their peers — in some instances, this might lead them to acknowledge that they were too positive or negative in their initial assessment; in others, they might be able to explain more clearly why they hold a particular point of view. In both cases, this provides valuable additional information to the editor, whose job it is to weigh up such differences of opinion and decide how to proceed. It’s hard to put numbers on how often cross-referee commenting might lead an editor to change his or her decision — I think this will be pretty rare — but I think it will quite frequently help the editor to provide better advice to the authors on how to revise their paper. So although we think cross-referee commenting will only have a significant impact in a minority of cases, we felt it would be fairest to introduce it across all papers.
RW: You said in your editorial that peer-review reports often read like “shopping lists.” What do you mean by this?
KB: I think there can be a tendency for referees to ask for too much — it’s always possible to think of additional experiments that authors could do. But in many cases these are experiments that would extend the study in various directions, rather than ensure that the current conclusions are sufficiently solid. So a review can sometimes ask the authors to do things that not only will take them many months, but also would likely form the basis of a whole new paper (or papers!) — rather than being a core requirement for this particular paper. In 2015, we changed our referee report form to try and avoid this. What we want referees to focus on is how well the data in a paper support the conclusions, rather than how the authors can make those conclusions more ‘interesting’. We think of this as making sure the referee limits his or her comments to the ‘necessary’ revisions rather than the ‘nice-to-have’s’. So in the shopping list analogy, the referee report should contain the milk, bread and vegetables (the things you need), but avoid the chocolate and wine (the luxuries)!
RW: How have authors and reviewers reacted to this announcement?
KB: The response to our announcement has been hugely positive, from both the Development and the Journal of Cell Science communities. We think this is because we’re making this change in response to their views — as a direct response to our survey. We’ll be rolling out the new process in the next week or two and then we will get to really see it in action. But we know from other journals that it works well and is generally popular.
RW: Will the new reviewing process at your journal affect referees’ option of remaining anonymous (by not signing their reports)? If so, how?
KB: This was something we thought hard about, and decided to retain anonymity: Unless referees choose to reveal their identities, they will remain anonymous to both the other referees and the authors. At this point, we don’t believe that open peer review would help to improve the process, and we suspect that some referees might decline to review if their identities would be known to other parties. Anecdotally, we’ve been told by referees that open discussions can be uncomfortable, and our survey reinforced the lack of enthusiasm for open peer review in our communities — this came out as one of the least popular potential innovations.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.