A new reproducibility fix? Get your work checked before it’s published

Most researchers by now recognize there’s a reproducibility crisis facing science. But what to do about it? Today in Nature, Jeffrey S. Mogil at McGill University and Malcolm R. Macleod at the University of Edinburgh propose a new approach: Restructure the reporting of preclinical research to include an extra “confirmatory study” performed by an independent lab, which verifies the findings before they are published. We spoke with them about how this could work.

Retraction Watch: You’re proposing to restructure animal studies of new therapies or ways to prevent disease. Can you explain what this new type of study should look like, and how researchers will execute it?

Jeffrey Mogil and Malcolm R. Macleod: We’re proposing not to restructure animal studies themselves, but rather restructure how those studies are published.  We recognize that in the preclinical domain, some experiments are clearly hypothesis-generating or hypothesis-supporting (i.e., “exploratory studies”), and others, typically the last study of a multi-experiment paper, are meant to be confirmatory.  We propose that the rules governing the former studies are lightened up (for example, by not requiring formal statistical analysis), but that at some point or another authors would need to “put up or shut up” and submit to a formal confirmatory study, performed with the highest levels of rigor by an independent lab (the second part). Both would be required for a paper. For studies that would provide the rationale for clinical trials in humans we suggest a third stage, a multicenter study.

RW: What do you hope to achieve with this change? How will it address the ongoing issues of reproducibility in preclinical work?

JM and MM: We see this idea as a compromise acceptable to both “sides” of the reproducibility issue:  the preclinical scientists who rightly see high levels of rigor imposed from above as threatening to their efficiency and flexibility, and stakeholders who need to trust the results of these preclinical studies.  With our plan, each side gets most of what it wants.  Preclinical scientists have increased freedom over the current status quo to explore in as efficient a manner as they please, but nothing actually gets published until it is confirmed not to be a false-positive result using levels of rigor that also exceed current practice.

RW: It seems from what you propose that the so-called “confirmatory studies” could be quite expensive – you say sample sizes for animal research could increase six-fold. Although collectively costs could decrease if researchers waste fewer resources following up on non-reproducible papers, to some individual researchers, the costs would rise. How do you propose addressing this?

JM and MM: Yes, the confirmatory studies are going to cost more money that typical final studies today.  But all the studies before that (and they are numerous in a typical high-impact paper) are going to cost less, because there will be no need to power each and every one of them to reach a p<0.05 statistical threshold.  And, of course, money will be saved in the system overall if this change leads to higher levels of reproducibility.  Who is going to pay for the confirmatory studies?  Either labs performing such studies would be fee-for-service and the experiment would be paid for out of the originating lab’s grant money, or funding organizations may (and should!) decide to fund high-quality confirmation labs to perform a certain number of these experiments a year as a service to others in the field.

RW: What are the main criticisms you’ve heard about your proposal, and how do you respond?

JM and MM: Some would prefer that rigor be tightened at every stage of preclinical research.  This strikes us as somewhat unreasonable, and likely to lead to resistance from the community. We believe our approach – of reinforcing rigor in those hypothesis-testing experiments where it is so important – is an approach more likely to succeed. Certain stakeholders are primarily interested in the reduction of false-positive findings, but preclinical scientists (and the public) are also quite rightly concerned with the reduction of false-negative findings as well (i.e., failing to identify that new cancer target that might slip through your preclinical research).  Most things that are done to reduce the former simply inflate the latter.  We believe our idea is one way to balance these competing demands.  Another reasonable question is how exactly this would be implemented?  Would journals or funding agencies make the idea mandatory?  And for whom exactly?  Some research isn’t easily characterized as requiring the kind of “preclinical trial” that we propose, but we believe that the principle – that hypothesis-testing experiments should do just that – has general applicability.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.