Retraction Watch

Tracking retractions as a window into the scientific process

“Failure is an essential part of science:” A Q&A with the author of a new book on reproducibility

with 9 comments

Reproducibility is everywhere recently, from the pages of scientific journals to the halls of the National Academy of Sciences, and today it lands in bookstores across the U.S. Longtime NPR correspondent Richard Harris has written Rigor Mortis (Basic Books), which is published today. (Full disclosure: I blurbed the book, writing that “Harris deftly weaves gripping tales of sleuthing with possible paths out of what some call a crisis.”) Harris answered some questions about the book, and the larger issues, for us. 

Retraction Watch (RW): Rigor Mortis begins with the story of the 2012 Nature paper by C. Glenn Begley and Lee Ellis that is now famous for sounding the alarm about reproducibility in basic cancer research. But as you document, this is not a problem that began in 2012. When did scientists first start realizing there was a problem?

Richard Harris
Credit: Meredith Rizzo

Richard Harris (RH): Some people cite John Ioannidis’s notable paper, published in 2005, titled “Why Most Published Research Findings are False,” but even in that paper he’s citing previous concerns. CK Gunsalus at the University of Illinois reaches back to Demosthenes (384-322 BC), who said “nothing is easier than self-deceit.” That’s clearly the nub of the problem. I’m actually not convinced it’s a crisis. What is new is scientists are increasingly aware of these serious problems. That’s actually good. Nobody wants science to spin its wheels, and recognizing a problem is the first step toward solving it.

RW: You’ve reported on science at NPR for three decades. How and when did you become interested in reproducibility?

RH: Throughout my career I’ve had queasy feelings about some of the studies that make headlines. But I didn’t really think of these problems as systemic until I read the Begley/Ellis paper. In 2014 I started looking at the overall health of the biomedical research enterprise and discovered how the funding roller-coaster was creating perverse incentives. That’s when it all clicked for me, cause and effect.

RW: The book highlights key stories not just of scientists struggling to reproduce their (and others’) findings, but of people like Lester “Randy” Curtin, who died of a rare brain tumor. (Curtin was also a statistician.) What effect does a lack of reproducibility have on patients?

RH: I remember being in the office of Anna Barker at Arizona State University and looking at a poster covered with print so small I could barely make it out. Barker told me those were the list of the 200 brain-cancer clinical trials, every one a failure. Most of the harm comes in this invisible form — diseases not cured, hopes not met. Occasionally people are actively harmed by these problems. For example, doctors prescribed hormone replacement therapy (estrogen and progestin) to postmenopausal women for many years before they realized they were actually causing thousands of excess cases of breast cancer and heart disease. The failures of basic biomedical research are more subtle but arguably have greater impact.

RW: We focus on retractions, as you’d expect, here at Retraction Watch, which means we’re often focused on misconduct. How much failure to reproduce do you think is due to misconduct?

RH: Misconduct in the legal sense is apparently responsible for a small fraction of problems, as far as I could discern, and as you have noted here on Retraction Watch. Social scientist Brian Martinson is much more concerned about what he calls “undesirable behaviors,” which is not blatant falsification or malfeasance, but more like corner-cutting — from sloppy handling of data to skipping repeat experiments. Scientists in the back of their minds know they probably shouldn’t be doing these things, but they may rationalize that it’s probably not affecting their results. These are far more common, given the hypercompetitive world of research, where the pressure is on not only to produce, but to produce quickly and to come up with flashy results.

RW: There has been some backlash against efforts to improve reproducibility. Robert Weinberg — who has had to retract five papers — refers in Rigor Mortis to the Reproducibility Project: Cancer Biology as filled with “silliness and naivete.” Some of the criticism even comes from people like Glenn Begley, who clearly thinks reproducibility is critical. How would you characterize scientists’ attitudes toward these issues in 2017?

RH: My sense is most scientists perceive there’s a problem (and that’s backed up by a 2016 poll in Nature). But while there’s a desire to make biomedical research more rigorous and reliable, there’s also a profound sense of unease, given the Trump administration’s budget plan, which calls for a 19 percent cut to the NIH budget. That would be a tsunami, and of course it would exacerbate the already fraught scramble for scarce research dollars. As best I can tell, that cut is not based on a rational analysis of the strengths and weaknesses of biomedical research, but scientists feel it’s an awkward time to be airing these issues.

RW: The last chapter of Rigor Mortis highlights potential ways to improve reproducibility. Which did you find the most compelling?

RH: There are a few simple steps, like requiring scientists to confirm the identity of the cells they use in research (the NIH now expects that of its grantees). There’s also a lot to be said for asking scientists to declare in advance what hypothesis they are testing (as happens in medical research when scientists register at ClinicalTrials.gov) and to make their ingredients, computer code and results readily available. Failure is an essential part of science, so the aim should be to reduce errors when possible, and to allow other scientists to find the ones you miss. The biggest challenge is to change the incentives that scientists face so they aren’t rewarded for cutting corners, massaging their data and making other decisions that may help their careers but which harm the scientific enterprise. That’s a tall order, I know.

Want to hear more from Harris and Ioannidis? I’ll be moderating a conversation between them at New York University tomorrow (April 5). For those not in New York, it will be livestreamed and archived here.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.

Written by Ivan Oransky

April 4th, 2017 at 9:23 am

Posted in not reproducible

Comments
  • fernandopessoa April 4, 2017 at 9:55 am

    “Trump administration’s budget plan, which calls for a 19 percent cut to the NIH budget.”
    and
    “scientists feel it’s an awkward time to be airing these issues.”

    Isn’t that a bit of scientific imperialism? Science is universal, not just confined to the U.S..
    Why do people think that airy issues about scientific reproducibility harms science, that is unless they are not really performing science in the first place.

  • Phillip J. Baker April 4, 2017 at 10:14 am

    I remember the many times, when I first joined the NIH in 1965, I would spend a good portion of the afternoon in the NIH Library, perusing articles in past issues of scientific journals. There is a vast difference — between then and today– in the quality of the writing. One could pick up any issue of the Journal of Immunology, for example, published prior to 1960, read it from cover-to-cover, and understand the purpose and significance of the work reported. Sadly, most of the article published today appear to be so specialized that they simply can not be written for more an extremely small audience to address an issue that of limited — if any– significance. Furthermore, most articles are so “loaded” with acronyms, Greek letter, and abbreviations that they almost appear to be written in code, all of which are obstacles to effective communication. At a minimum, editorial boards need to demand that authors explain why the work is being done and its significant with respect to addressing an issue of broad import in the context of previous studies, as well as how the work was performed to achieve it objectives. Is it asking too much that this be done in clear and simple English?

    • fernandopessoa April 4, 2017 at 10:44 am

      “Is it asking too much that this be done in clear and simple English?”

      Most likely.

  • Gaythia Weis April 4, 2017 at 10:19 am

    I’d like to see much more institutional support for lower level researchers, grad students, and other potential whistleblowers. It is the larger culture that needs changing. “The biggest challenge is to change the incentives that scientists face so they aren’t rewarded for cutting corners, massaging their data and making other decisions that may help their careers but which harm the scientific enterprise.”

    • µ April 5, 2017 at 12:29 am

      Well said! Agree

  • Donald S Kornfeld April 4, 2017 at 11:18 am

    Collins and Tabak declare there’s a,crisis in reproducibility, based on two papers (Begley/Prinz) which report that work submitted by outside labs could not be replicated by pharmaceutical companies. Do two such reports make a crisis ?

    They also dismiss the contribution of research misconduct based on the small annual number of guilty investigators.They ignore an earlier paper by two ORI
    investigators who estimated that 1,000 cases of research misconduct are not reported annually.Almost 50% of retracted papers were based on research misconduct.

    It is definitely easier to require better training in statistics and variability of biological samples then enact policies which address that more serious problem.

    Don Kornfeld

    • Jonathan L. Seagull April 5, 2017 at 6:51 pm

      I recommend reading the book that this post is advertising, or listening to the talk that is going on right now.

  • firststone April 4, 2017 at 11:53 pm

    That’s the tall order, as they said. The culture needs changing. Anything short of that will prove to fail.

    Gaythia Weis
    I’d like to see much more institutional support for lower level researchers, grad students, and other potential whistleblowers. It is the larger culture that needs changing. “The biggest challenge is to change the incentives that scientists face so they aren’t rewarded for cutting corners, massaging their data and making other decisions that may help their careers but which harm the scientific enterprise.”

  • Michael April 5, 2017 at 7:59 pm

    I’d love to see Trump administration not cut NIH funding, but appropriate and redirect sufficient funds to the ORI, increase its staff, and let the FBI handle the investigations (instead of asking Universities to self-police; with the usual fox-being-asked-to-investigate-problems-in-the-hen-house scheme), and dispatch those crooked among us to the big-house for time commensurate with the federal funds misused and patients harmed.

  • Post a comment

    Threaded commenting powered by interconnect/it code.