David Vaux, a cell biologist at the Walter + Eliza Hall Institute of Medical Research in Melbourne, explains how Nature could do more to remove bias from the peer review process. He previously wrote about his decision to retract a paper.
Last week, Nature announced that they are to offer authors of papers submitted to Nature or the monthly Nature research journals the option of having their manuscripts assessed by double-blind peer review, in which reviewers are blinded to the identity of authors and their institutions. Until now, papers sent to Nature, and most other journals, have been reviewed by a single-blind process, in which the reviewers know the identities and affiliations of the authors, but the authors are not told who the reviewers are. The goal of double-blind peer review is for submitted papers to be judged on their scientific merit alone, and thus to reduce publication bias.
While Nature should be applauded for this move, the way they have implemented it leaves room for improvement.
Science is built on objective evidence, but, because it is a human endeavor, prejudices, biases and subjectivity inevitably creep in.
This problem has been tackled well by clinical studies, in which the investigators are blinded, the subjects are not told whether they are receiving the active treatment or a placebo, and the treatments are randomized. The widespread adoption of double-blind clinical trials was one of the greatest scientific advances of the last century. Open testing, in which both the patients and the clinicians know who is getting the active drug, gives more transparency, but is only used in early phase trials when the dose of the drug is being determined. Even clinicians who considered themselves to be honest and free from bias would not advocate non-blinded testing in phase III – not because they think that they are dishonest, but because they know that unconscious biases and prejudices are impossible to eliminate.
While double-blinding is standard in clinical trials, there is currently debate about the best way to peer review. Most journals have a single-blinded system, in which the reviewers know who the authors are, but the authors don’t know who the reviewers are. The weakness of this system is that reviewers’ opinions can be influenced by the reputations of the authors or their affiliations. For example, they might less readily dismiss a surprising finding from a Nobel Laureate than if the same finding was reported by an unknown researcher from Biddelonia.
It was while reading Nature that I first started to worry about the peer review process. Scanning the 19 February 2004 edition, I found several papers that had figures with error bars, but most didn’t say what the error bars were, either in the legends or the Materials and Methods.
If the editors or reviewers had noticed these mistakes, I presume they would have asked for them to be corrected. Because they were not, it appeared that the papers had not been read thoroughly, either by the reviewers, the editors, or the authors. Presumably the decision to accept the papers was based on something that could be ascertained without reading the manuscript – such as the identity or affiliation of the authors. (Lest anyone believe this is a problem only in Nature, it’s not.)
In response to my letter, Nature improved their instructions to authors and reviewers, reminding them to check the descriptions of the error bars. As a result, papers containing figures with undescribed error bars appear much less frequently. I, however, continue to worry, because these reminders might have been treating the symptom — i.e. failure to describe error bars — rather than the underlying disease: The failure of authors, reviewers and editors to read papers carefully.
Double-blind peer review would help, by forcing reviewers to read the paper before deciding whether to accept or reject it.
Another proposal to improve peer review is to do the opposite: make it entirely unblinded, so that the reviewers’ names are revealed to the authors, and their comments are published along with the authors’ paper. Open peer review discourages reviewers from writing superficial reviews, and from including ad hominem attacks, but it does suffer from two weaknesses. Firstly, junior researchers might be discouraged from making comments that are critical of powerful and influential authors, and secondly, the community will only ever see the reviews for papers that are eventually accepted.
Post-publication peer review is yet another option. In its purest form, all papers are published, and then readers add comments, and the authors change their papers on-line, or defend what they have previously written. A disadvantage of post-publication review is that if used in isolation, it loses the filtering of papers provided by pre-publication review. It would also be difficult to prevent gaming the system by authors who commented on their own papers.
Double-blind peer review has also been criticized. It has often been pointed out that by reading a paper thoroughly, and going over the Materials and Methods and the References, it may be possible to guess who the authors might be. While that may be true, if reviewers have to read the paper that carefully, then the battle is half won. Double-blind review might even encourage authors to cite other laboratories more frequently, so as to put reviewers off the scent.
Nature plans to make double-blind review optional. This is a major flaw, as high profile authors from famous institutions might choose single-blind review, in order to impress the reviewers. On the other hand, when reviewers receive blinded manuscripts, they might assume that they are from obscure authors from lowly institutions.
Nature’s plan might also fail if the authors’ identities are not withheld from the editors who make the decision whether to send a paper out for review, who are also vulnerable to bias. Of course, when selecting the reviewers, measures would have to be taken to prevent the blinded editors from choosing the authors to review their own papers, but this should not be too difficult to implement. Editors who think a paper should be sent for review could come up with a list of six potential reviewers with different affiliations, and give the list to another staff member who would check the list and delete conflicted reviewers.
Of the different types of peer review, double-blind review has proven the most popular in two large surveys of researchers, the 2009 survey reported by Mulligan et al. (rated it most “effective”), and the survey by Mark Ware of The Publishing Research Consortium. The barrier to adoption of double-blind review therefore seems to come from editors and publishers, rather than the authors.
I hope Nature’s experiment is successful. I encourage them to make double-blind review mandatory, and to blind the editors as well as the reviewers. Other measures to improve peer review would be to screen images in figures for adherence to guidelines, and to link the reviewers’ (anonymous) comments and authors’ rebuttals to the online versions of the papers. Enabling post-publication review, either on the journal’s site or by linking to another site (such as PubMed Commons) would allow papers to be corrected and improved more efficiently.
Peer review is a vital part of how we do science, and to experiment with different forms is a laudable, and scientific, way forward, for which Nature should be congratulated.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.
Double-blind peer review does not work in the age of preprint servers, where it’s trivial to find out who actually wrote the paper.
What if they stared by randomly assigning papers to the double blind or traditional process? Then they could compare the outcomes. Even a short experiment would start to uncover whether prestige was influencing the peer reviewers. After years pass, it should start to be possible to compare the quality of the papers from different tracks – which are retracted more often, etc.
If results from the double blind process are promising, they could phase it in.
Excellent points raised! Though Nature should be applauded for this move, I think several areas (i.e., computer science, at least) have used double-blind review in conferences for at least 10+ years. So, we shouldn’t assume that what Nature is doing is new — it’s just getting more attention. Their solution to self-citing is to ask authors to be aware of their writing and not cite too much of their own work and not make silly mistakes like saying, “In our previous work…”. If authors can do this, then maybe reviewers can reciprocate by not purposely trying to use Google to figure out who the authors are?
Also a good point about making editors blind and having double-blind being mandatory.
But, I have to disagree the part about making it entirely open. That is, by making the names of reviewers open to authors. It is a good idea, in the name of science. But we should also be aware of the personal safety of the reviewers — this might range from future biases in reviewing their work or even harassment. Sadly, some scientists may act emotional after receiving their review. The reverse can also happen — could giving a positive review make it easier for me to apply for a position with the lab that wrote the paper?
I hope they give double-blind reviewing a chance — it seems like a step in the right direction.
In addition to the problems already raised, there is yet another one. In fields where the number of active people is small,
there would be little problem in identifying authors by looking at the methods they use, and by checking the list of references.
By the same token, it’s often pretty obvious who the referees are.
Pre-publication peer review has so often proved to be unsatisfactory that it seems to me to be inevitable that post-publication review is the future,
This sounds reminiscent of an experiment described by Malcolm Gladwell about choosing new orchestra members for a German orchestra. Women were lacking in representation, so the “blind review”there was to have musicians audition behind a screen so their gender, age, ethnicity were not apparent. All of a sudden women were selected along with men. Blind selection does help screen for bias. I like the idea of double-blind review.
I did some research for my editorial in Laborjournal (http://laborjournal.de/editorials/918.lasso), apparently the evidence for gender discrimination in journal peer review (as opposed to funding grants and positions) is rather thin:
http://onlinelibrary.wiley.com/doi/10.1002/asi.22784/full
“Although empirical research on gender bias in publication and grant outcomes has produced “data and interpretations which at times are contradictory” (Rees, 2011, p. 140), recent meta-analysis suggests that claims of gender bias in peer review “are no longer valid” (Ceci & Williams, 2011, p. 3,157).”
A step in the right direction, but the step in itself will have no positive effect. As was already mentioned, high profile authors will always use their name. Those that submit blinded will be assumed to be no-name authors and their work untrustworthy.
I have to agree with Bernd. Traditional single-blind or double-blind peer review is now, as of 2015, old fashioned. Open peer review is in. Nature lost its chance and it only presented this inadequate band-aid to save face and try and reduce criticisms and contain damage control caused by what appears to be an increasingy critical PPPR pool, and increasing retractions. It doesn’t matter if editors and peer know the authors’ identitities and vice versa, because ultimately the paper will be out there, open to public scrutiny. The sooner all publisher start to move away from an exclusive traditional peer review system in whatever blind format, the better. There is only one realistic solution: a three-tier system consisting of a pre-submission peer review, a traditional peer (double-blind) and an open-ended open peer system that allows for ad infinitum commenting and critiquing, through PPPR. A dreadful change, no doubt, but an absolutely essential one. That will begin to differentiate the wheat from the chaff, and wil undoubtedly lead to a massively bloated pool of “predatory” publishers as the class of classy publishers becomes more restricted. So, my message to NPG/McMillan, enough of band-aids, how about some real reform, especially as you merge with Springer.
Double-blind review is nothing but a gimmick. In some relatively small fields it is very easy to guess the identity of the reviewer – based on a relatively short review. Given a full-length manuscript the identity of the corresponding author(s) becomes just about transparent. Newcomers excluded, obviously. In larger and more fluid fields a fairly simple search would yield the identity of the author(s) or reveal the fact that the author is a nobody (lots of them about).
The more open is the review process the better. And Nature coming up with this blind spot but refusing to make pdb validation reports mandatory for structural papers – which is essential to assist reviewers in evaluating the quality of work rather than Pymol/Chimera-generated pictures?! Gimmick.
So much about publishing in Nature depends on whether an editor even sends out a paper for peer review , and who the reviewers are. Nothing about the editorial process will change under this system, so the high profile labs still have their mediocre stuff sent out for review, and the others get editorially rejected.
The most important action that needs to be taken is to blind the editors to the identity of the authors. 90% or more of manuscripts submitted to Nature are rejected editorially. The scientists who for some reason (such as being winded and dined by them) are considered by the editors as the leaders in the field will NEVER get their manuscripts editorially rejected. Nature should be commended for this innovation but they should be encouraged to treat all manuscripts in a double-blinded manner and blind the editors to the identity of the authors.
“Ending is better than mending”
-Aldous Huxley, Brave New World
Another more radical form of blinding is getting reviews of methods WITHOUT results. There’s some evidence that reviewers are biased by the direction of results (ie positive or negative, agreeing with their views or not, etc.). I understand the journal, Cortex, has introduced ‘registered reports’ which means the protocol is reviewed and the journal then guarantees to publish the findings. The Lancet introduced protocol review some years ago but without the guarantee to publish, so it didn’t really catch on.
If the question is important and the methods are sound, then the research should be published. When reviewing protocols (at the start of a study) it might also be easier to hide the identity and affiliation of authors (in some cases), so this might also facilitate this type of blinding.
Another advantage of this system is that if reviewers spot a major weakness in the methods, there is still time for the researchers to correct this.
Double blind is not enough. Many journals have implemented such double blind methods. Triple blind method will give better democracy. The administrators (like even the website) should not also know the names and affiliations of the authors. If any one knows then everybody knows. Moreover papers should be written in such a way that no one can even guess who the authors are from reading the paper and references. If such detection becomes meaningful then the paper must be rejected.
Democracy and capitalism do not go together. So such a triple blind method will not be acceptable by reputable institutions and business houses. It will affect their business goals. So methods will be created to defeat the system. After all nothing can be secret in the digital age. A little effort will uncover the identity of author names and their affiliations.
In all the subfields I publish in (none of them in the “hard sciences”; one or maybe two in pure mathematics, one in computer science, one in mathematical psychology), that would range from nearly to completely impossible. Would you allow an accepted paper to be re-written to restore necessary (or at least very useful) information that would have had to be omitted under your proposal? (Some conferences in robotics do allow this, but they’re only double-blinded.)
As you mention, some areas within computer science do allow information to be restored in an accepted paper.
Some of these conferences also ask authors to write papers in such a way so that their identity cannot be “easily” guessed. It would be feasible to have such papers immediately rejected if the authors have not done their part in writing their manuscript anonymously. Similarly, there is some expectation that even though reviewers are human, that they will refrain from entering information from the paper into Google to try to figure out whose paper they are reviewing.
I think such an approach is worth trying but to succeed, both sides need to co-operate.
Other approaches proposed by others above sound “better”, but they also require such co-operation. For example, a completely open reviewing system requires both parties to not take any criticism personally or reward the other person favorably.
Since no review system can take away the parts that make us all human, no system is perfect (of course!). But the human element is always there, unless papers are submitted to a black box surrounding a 50/50 coin toss… (I think some dubious conferences currently use this approach!)
Going back from mathematical robotics (to which I am a relative newcomer) to pure mathematics (I got my Ph.D. in 1974), I am less concerned with the hope that reviewers will refrain from Google searches than I am with the fact that when a subfield is relatively new and/or relatively small, it may simply be impossible to find referees who are (1) competent to judge a paper, and (2) unaware of who the author most likely is. As to (1), of course there will always be some very top people in a larger field containing the subfield who are omnicompetent—but they’re always swamped with work and cannot reasonably be asked to do a huge amount of refereeing, too. As to (2), I’m now at the point where many of the people who picked up on a subfield that I may not have started, exactly, but where I was an early investigator (and in which I did have several good new ideas that have stood the test of time), none of whom were my students (because I’ve never had doctoral students) but all of whom I’ve met at meetings (some very often), have not only had students (many of whom I’ve met) who made progress by using/extending my method, their students are beginning to have such students! That is, 40 years on, the subfield is big enough that guessing is no longer easy. But 20 years ago, it wouldn’t have been hard for a referee to guess a blinded author; and 30 years ago, any referee could have known that a paper in the subfield was mine—there just could have been no way for me to mask myself.
I understand that mathematics is an outlier in various ways (shelf-life of articles, for instance: my two most cited articles are from 1983 and 1993, and the rate of citation is nearly constant; comparatively small size of the field as a whole, relative to bioscience, say; even smaller sizes of subfields; maybe others), but I find it hard to believe that both (1) and (2) are entirely irrelevant in other fields.