A new reproducibility fix? Get your work checked before it’s published

Jeffrey Mogil
Malcolm Macleod

Most researchers by now recognize there’s a reproducibility crisis facing science. But what to do about it? Today in Nature, Jeffrey S. Mogil at McGill University and Malcolm R. Macleod at the University of Edinburgh propose a new approach: Restructure the reporting of preclinical research to include an extra “confirmatory study” performed by an independent lab, which verifies the findings before they are published. We spoke with them about how this could work.

Retraction Watch: You’re proposing to restructure animal studies of new therapies or ways to prevent disease. Can you explain what this new type of study should look like, and how researchers will execute it?

Jeffrey Mogil and Malcolm R. Macleod: We’re proposing not to restructure animal studies themselves, but rather restructure how those studies are published.  We recognize that in the preclinical domain, some experiments are clearly hypothesis-generating or hypothesis-supporting (i.e., “exploratory studies”), and others, typically the last study of a multi-experiment paper, are meant to be confirmatory.  We propose that the rules governing the former studies are lightened up (for example, by not requiring formal statistical analysis), but that at some point or another authors would need to “put up or shut up” and submit to a formal confirmatory study, performed with the highest levels of rigor by an independent lab (the second part). Both would be required for a paper. For studies that would provide the rationale for clinical trials in humans we suggest a third stage, a multicenter study.

RW: What do you hope to achieve with this change? How will it address the ongoing issues of reproducibility in preclinical work?

JM and MM: We see this idea as a compromise acceptable to both “sides” of the reproducibility issue:  the preclinical scientists who rightly see high levels of rigor imposed from above as threatening to their efficiency and flexibility, and stakeholders who need to trust the results of these preclinical studies.  With our plan, each side gets most of what it wants.  Preclinical scientists have increased freedom over the current status quo to explore in as efficient a manner as they please, but nothing actually gets published until it is confirmed not to be a false-positive result using levels of rigor that also exceed current practice.

RW: It seems from what you propose that the so-called “confirmatory studies” could be quite expensive – you say sample sizes for animal research could increase six-fold. Although collectively costs could decrease if researchers waste fewer resources following up on non-reproducible papers, to some individual researchers, the costs would rise. How do you propose addressing this?

JM and MM: Yes, the confirmatory studies are going to cost more money that typical final studies today.  But all the studies before that (and they are numerous in a typical high-impact paper) are going to cost less, because there will be no need to power each and every one of them to reach a p<0.05 statistical threshold.  And, of course, money will be saved in the system overall if this change leads to higher levels of reproducibility.  Who is going to pay for the confirmatory studies?  Either labs performing such studies would be fee-for-service and the experiment would be paid for out of the originating lab’s grant money, or funding organizations may (and should!) decide to fund high-quality confirmation labs to perform a certain number of these experiments a year as a service to others in the field.

RW: What are the main criticisms you’ve heard about your proposal, and how do you respond?

JM and MM: Some would prefer that rigor be tightened at every stage of preclinical research.  This strikes us as somewhat unreasonable, and likely to lead to resistance from the community. We believe our approach – of reinforcing rigor in those hypothesis-testing experiments where it is so important – is an approach more likely to succeed. Certain stakeholders are primarily interested in the reduction of false-positive findings, but preclinical scientists (and the public) are also quite rightly concerned with the reduction of false-negative findings as well (i.e., failing to identify that new cancer target that might slip through your preclinical research).  Most things that are done to reduce the former simply inflate the latter.  We believe our idea is one way to balance these competing demands.  Another reasonable question is how exactly this would be implemented?  Would journals or funding agencies make the idea mandatory?  And for whom exactly?  Some research isn’t easily characterized as requiring the kind of “preclinical trial” that we propose, but we believe that the principle – that hypothesis-testing experiments should do just that – has general applicability.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.

5 thoughts on “A new reproducibility fix? Get your work checked before it’s published”

  1. This can only be a good thing for patients who are eagerly waiting therapies from VERY promising preclinical work. We, as yet, do not know why most promising preclinical therapeutics fail, often miserably, in human trials.

    I would also suggest, perhaps, contracting out to CROs if no suitable independent academic laboratory can be utilised.

  2. I would imagine that the lab running the confirmatory analyses would need to be selected judiciously. If people are organizing peer review rings to get favorable reviews then they could certainly do the same for confirmatory analyses.

    Also, separate point: Should there be an outlet for ‘working’ results? I.e., should there be some way to share results that haven’t been independently confirmed yet? Might promote more willingness from the field if people are able to share results sooner. Of course, such work would need to be labeled exploratory and interpreted with extreme caution.

    1. “Should there be an outlet for ‘working’ results?” – you’re talking about preprints. There’s increasing interest in them at the moment (for disclosure, I help run http://www.preprints.org). It has worked well in physics for a long time and now other disciplines are getting very interested, see ASAPbio (http://asapbio.org) as probably the most organised example.

    2. Are there any journals willing to ONLY publish preclinical data that HAVE been confirmed by an independent laboratory? I would suggest such a journal may rapidly become the GOLD standard for preclinical publications.

      Having previously been at the interface between academia and industry I would presume to state industrialists, who want bench-to-bedside therapeutics (rather than bench-to-failed-human-trial-to-bin), would certainly collaborate more readily with independently confirmed findings, rather than the present, potentially contaminated, peer review approvals.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.