“Failure is an essential part of science:” A Q&A with the author of a new book on reproducibility

Reproducibility is everywhere recently, from the pages of scientific journals to the halls of the National Academy of Sciences, and today it lands in bookstores across the U.S. Longtime NPR correspondent Richard Harris has written Rigor Mortis (Basic Books), which is published today. (Full disclosure: I blurbed the book, writing that “Harris deftly weaves gripping tales of sleuthing with possible paths out of what some call a crisis.”) Harris answered some questions about the book, and the larger issues, for us. 

Retraction Watch (RW): Rigor Mortis begins with the story of the 2012 Nature paper by C. Glenn Begley and Lee Ellis that is now famous for sounding the alarm about reproducibility in basic cancer research. But as you document, this is not a problem that began in 2012. When did scientists first start realizing there was a problem?

Richard Harris (RH): Some people cite John Ioannidis’s notable paper, published in 2005, titled “Why Most Published Research Findings are False,” but even in that paper he’s citing previous concerns. CK Gunsalus at the University of Illinois reaches back to Demosthenes (384-322 BC), who said “nothing is easier than self-deceit.” That’s clearly the nub of the problem. I’m actually not convinced it’s a crisis. What is new is scientists are increasingly aware of these serious problems. That’s actually good. Nobody wants science to spin its wheels, and recognizing a problem is the first step toward solving it.

RW: You’ve reported on science at NPR for three decades. How and when did you become interested in reproducibility?

RH: Throughout my career I’ve had queasy feelings about some of the studies that make headlines. But I didn’t really think of these problems as systemic until I read the Begley/Ellis paper. In 2014 I started looking at the overall health of the biomedical research enterprise and discovered how the funding roller-coaster was creating perverse incentives. That’s when it all clicked for me, cause and effect.

RW: The book highlights key stories not just of scientists struggling to reproduce their (and others’) findings, but of people like Lester “Randy” Curtin, who died of a rare brain tumor. (Curtin was also a statistician.) What effect does a lack of reproducibility have on patients?

RH: I remember being in the office of Anna Barker at Arizona State University and looking at a poster covered with print so small I could barely make it out. Barker told me those were the list of the 200 brain-cancer clinical trials, every one a failure. Most of the harm comes in this invisible form — diseases not cured, hopes not met. Occasionally people are actively harmed by these problems. For example, doctors prescribed hormone replacement therapy (estrogen and progestin) to postmenopausal women for many years before they realized they were actually causing thousands of excess cases of breast cancer and heart disease. The failures of basic biomedical research are more subtle but arguably have greater impact.

RW: We focus on retractions, as you’d expect, here at Retraction Watch, which means we’re often focused on misconduct. How much failure to reproduce do you think is due to misconduct?

RH: Misconduct in the legal sense is apparently responsible for a small fraction of problems, as far as I could discern, and as you have noted here on Retraction Watch. Social scientist Brian Martinson is much more concerned about what he calls “undesirable behaviors,” which is not blatant falsification or malfeasance, but more like corner-cutting — from sloppy handling of data to skipping repeat experiments. Scientists in the back of their minds know they probably shouldn’t be doing these things, but they may rationalize that it’s probably not affecting their results. These are far more common, given the hypercompetitive world of research, where the pressure is on not only to produce, but to produce quickly and to come up with flashy results.

RW: There has been some backlash against efforts to improve reproducibility. Robert Weinberg — who has had to retract five papers — refers in Rigor Mortis to the Reproducibility Project: Cancer Biology as filled with “silliness and naivete.” Some of the criticism even comes from people like Glenn Begley, who clearly thinks reproducibility is critical. How would you characterize scientists’ attitudes toward these issues in 2017?

RH: My sense is most scientists perceive there’s a problem (and that’s backed up by a 2016 poll in Nature). But while there’s a desire to make biomedical research more rigorous and reliable, there’s also a profound sense of unease, given the Trump administration’s budget plan, which calls for a 19 percent cut to the NIH budget. That would be a tsunami, and of course it would exacerbate the already fraught scramble for scarce research dollars. As best I can tell, that cut is not based on a rational analysis of the strengths and weaknesses of biomedical research, but scientists feel it’s an awkward time to be airing these issues.

RW: The last chapter of Rigor Mortis highlights potential ways to improve reproducibility. Which did you find the most compelling?

RH: There are a few simple steps, like requiring scientists to confirm the identity of the cells they use in research (the NIH now expects that of its grantees). There’s also a lot to be said for asking scientists to declare in advance what hypothesis they are testing (as happens in medical research when scientists register at ClinicalTrials.gov) and to make their ingredients, computer code and results readily available. Failure is an essential part of science, so the aim should be to reduce errors when possible, and to allow other scientists to find the ones you miss. The biggest challenge is to change the incentives that scientists face so they aren’t rewarded for cutting corners, massaging their data and making other decisions that may help their careers but which harm the scientific enterprise. That’s a tall order, I know.

Want to hear more from Harris and Ioannidis? I’ll be moderating a conversation between them at New York University tomorrow (April 5). For those not in New York, it will be livestreamed and archived here.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.