David Vaux, a cell biologist at the Walter + Eliza Hall Institute of Medical Research in Melbourne, explains how Nature could do more to remove bias from the peer review process. He previously wrote about his decision to retract a paper.
Last week, Nature announced that they are to offer authors of papers submitted to Nature or the monthly Nature research journals the option of having their manuscripts assessed by double-blind peer review, in which reviewers are blinded to the identity of authors and their institutions. Until now, papers sent to Nature, and most other journals, have been reviewed by a single-blind process, in which the reviewers know the identities and affiliations of the authors, but the authors are not told who the reviewers are. The goal of double-blind peer review is for submitted papers to be judged on their scientific merit alone, and thus to reduce publication bias.
While Nature should be applauded for this move, the way they have implemented it leaves room for improvement.
Science is built on objective evidence, but, because it is a human endeavor, prejudices, biases and subjectivity inevitably creep in.
This problem has been tackled well by clinical studies, in which the investigators are blinded, the subjects are not told whether they are receiving the active treatment or a placebo, and the treatments are randomized. The widespread adoption of double-blind clinical trials was one of the greatest scientific advances of the last century. Open testing, in which both the patients and the clinicians know who is getting the active drug, gives more transparency, but is only used in early phase trials when the dose of the drug is being determined. Even clinicians who considered themselves to be honest and free from bias would not advocate non-blinded testing in phase III – not because they think that they are dishonest, but because they know that unconscious biases and prejudices are impossible to eliminate.
While double-blinding is standard in clinical trials, there is currently debate about the best way to peer review. Most journals have a single-blinded system, in which the reviewers know who the authors are, but the authors don’t know who the reviewers are. The weakness of this system is that reviewers’ opinions can be influenced by the reputations of the authors or their affiliations. For example, they might less readily dismiss a surprising finding from a Nobel Laureate than if the same finding was reported by an unknown researcher from Biddelonia.
It was while reading Nature that I first started to worry about the peer review process. Scanning the 19 February 2004 edition, I found several papers that had figures with error bars, but most didn’t say what the error bars were, either in the legends or the Materials and Methods.
If the editors or reviewers had noticed these mistakes, I presume they would have asked for them to be corrected. Because they were not, it appeared that the papers had not been read thoroughly, either by the reviewers, the editors, or the authors. Presumably the decision to accept the papers was based on something that could be ascertained without reading the manuscript – such as the identity or affiliation of the authors. (Lest anyone believe this is a problem only in Nature, it’s not.)
In response to my letter, Nature improved their instructions to authors and reviewers, reminding them to check the descriptions of the error bars. As a result, papers containing figures with undescribed error bars appear much less frequently. I, however, continue to worry, because these reminders might have been treating the symptom — i.e. failure to describe error bars — rather than the underlying disease: The failure of authors, reviewers and editors to read papers carefully.
Double-blind peer review would help, by forcing reviewers to read the paper before deciding whether to accept or reject it.
Another proposal to improve peer review is to do the opposite: make it entirely unblinded, so that the reviewers’ names are revealed to the authors, and their comments are published along with the authors’ paper. Open peer review discourages reviewers from writing superficial reviews, and from including ad hominem attacks, but it does suffer from two weaknesses. Firstly, junior researchers might be discouraged from making comments that are critical of powerful and influential authors, and secondly, the community will only ever see the reviews for papers that are eventually accepted.
Post-publication peer review is yet another option. In its purest form, all papers are published, and then readers add comments, and the authors change their papers on-line, or defend what they have previously written. A disadvantage of post-publication review is that if used in isolation, it loses the filtering of papers provided by pre-publication review. It would also be difficult to prevent gaming the system by authors who commented on their own papers.
Double-blind peer review has also been criticized. It has often been pointed out that by reading a paper thoroughly, and going over the Materials and Methods and the References, it may be possible to guess who the authors might be. While that may be true, if reviewers have to read the paper that carefully, then the battle is half won. Double-blind review might even encourage authors to cite other laboratories more frequently, so as to put reviewers off the scent.
Nature plans to make double-blind review optional. This is a major flaw, as high profile authors from famous institutions might choose single-blind review, in order to impress the reviewers. On the other hand, when reviewers receive blinded manuscripts, they might assume that they are from obscure authors from lowly institutions.
Nature’s plan might also fail if the authors’ identities are not withheld from the editors who make the decision whether to send a paper out for review, who are also vulnerable to bias. Of course, when selecting the reviewers, measures would have to be taken to prevent the blinded editors from choosing the authors to review their own papers, but this should not be too difficult to implement. Editors who think a paper should be sent for review could come up with a list of six potential reviewers with different affiliations, and give the list to another staff member who would check the list and delete conflicted reviewers.
Of the different types of peer review, double-blind review has proven the most popular in two large surveys of researchers, the 2009 survey reported by Mulligan et al. (rated it most “effective”), and the survey by Mark Ware of The Publishing Research Consortium. The barrier to adoption of double-blind review therefore seems to come from editors and publishers, rather than the authors.
I hope Nature’s experiment is successful. I encourage them to make double-blind review mandatory, and to blind the editors as well as the reviewers. Other measures to improve peer review would be to screen images in figures for adherence to guidelines, and to link the reviewers’ (anonymous) comments and authors’ rebuttals to the online versions of the papers. Enabling post-publication review, either on the journal’s site or by linking to another site (such as PubMed Commons) would allow papers to be corrected and improved more efficiently.
Peer review is a vital part of how we do science, and to experiment with different forms is a laudable, and scientific, way forward, for which Nature should be congratulated.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.