So how often does medical consensus turn out to be wrong?
In a quote that has become part of medical school orientations everywhere, David Sackett, often referred to as the “father of evidence-based medicine,” once famously said:
Half of what you’ll learn in medical school will be shown to be either dead wrong or out of date within five years of your graduation; the trouble is that nobody can tell you which half–so the most important thing to learn is how to learn on your own.
Sackett, we are fairly sure, was making an intentionally wild estimate when he said “half.” [See note about these strikethroughs at bottom of post.]
But aA fascinating study out today in the Archives of Internal Medicine gives a clue as to the real figure suggests that he may have been closer than any of us imagined.
The authors of the study, from Northwestern and the University of Chicago, looked at a year’s worth of studies in the New England Journal of Medicine (NEJM) that “made some claim with respect to a medical practice.” There were 124 such studies, and 16 — that’s 13 percent, or about one in 8 — constituted reversals. As the authors note:
Reversals included medical therapies (prednisone use among preschool-aged children with viral wheezing, tight glycemic control in intensive care unit patients, and the routine use of statins in hemodialysis patients), invasive procedures (endoscopic vein harvesting for coronary artery bypass graft surgery and percutaneous coronary intervention for chronic total artery occlusions and atherosclerotic renal artery disease), and screening tests. In several cases, current guidelines were contradicted by the study in question, as indicated in the third column of the eFigure.
So a quick back-of-the-envelope calculation suggests that at 13 percent reversals per year during four years of medical school, for 52 percent, Sackett may have been remarkably close to reality. Then again, that makes a lot of assumptions, including ones about how representative this sample is, which we’ll get to in a moment. Still, as long as that one in eight figure is in the ballpark — and there’s reason to think it is — tThe results mean that a good chunk of current medical practice is likely to be reversed over time. And although that doesn’t mean all of those original papers should be retracted, you can see our obvious interest in this.
We asked study co-author Vinay Prasad, by email, to elaborate on the findings. First, what made the team decide to do this analysis:
For a long time, we were interested by what we believe to be a pervasive problem in modern medicine. Namely, the spread of new technologies and therapies without clear evidence that they work, which are later (and often after considerable delay) followed by contradictions, which, in turn, after yet another delay, is followed by changes in practice and reimbursement.
One might contend that if the cardinal principle of medicine is ‘first do no harm’– reversal violates this. First, there is harm to the patients who underwent the therapy during the years it fell in favor, and second, the harm to future patients until there is a change in practice. And lastly, there are the diffuse harms, such as loss of trust in the medical system. The USPSTF’s change of mammography for 40 year old women had painful repercussions, and was based in large part on the Lancet reversal in 2006.
Prasad noted that there had been an earlier study in JAMA of contradictions in medical research by John Ioannidis, whose work on the shoddy state of clinical evidence has been getting more attention lately:
Ioannidis shows that among highly cited research 16% of findings are later contradicted. There were two limits to Ioannidis paper, however, despite its merits. The first is that high citation papers may overrepresent controversy, as controversial topics draw further citations and discussion. Although, to be fair, this is likely a limit to our paper.
The second reason is that we were specifically interested in what percentage of standard of care is ultimately mistaken, which is different than what percent of high citation papers are later contradicted. The former represents what doctors actually do, while high citation papers may not necessarily reflect clinical practice.
Naturally, we wondered whether the results could be generalized to journals other than the NEJM:
There are reasons to believe we are over and underrepresenting reversal. So the answer is we don’t know.
Overrepresent: The NEJM probably gets more reversals than other journals because good reversals are large randomized trials, which are highly coveted by high impact journals (Lancet, NEJM, JAMA). So reversal may not be the same for other journals, particularly lesser ones.
Underrepresent: However, on the other hand, for all the possible testable questions (that arise from current clinical practice) only a fraction are being tested at any given time. The bulk of NEJM is evaluating new therapies (72%, in our paper) as opposed to established ones. There are likely more reversals out there. Further work is warranted, and we have some ideas on how to extend our analysis.
So how long do these kinds of reversals typically take?
So, no one has looked at how long practices survive before they are ‘reversed.’ But, I would argue that it has changed over the years, and accelerated since the [Cardiac Arrhythmia Suppression Trial] CAST trial in the early 1990s. Reversal probably happens faster now (though still pretty slow). The Nesiritide study in this week’s NEJM has a nice editorial by Eric Topol, who talks about a ‘lost decade,’ i.e. how long nesiritide was used before being contradicted. But, examples like routine use of the pulmonary artery catheter (one of the early reversals) took decades before a solid reversal (ESCAPE trial in 2005).
And do these reversals have an impact? If so, how long do they tend to take to change practice?
Yes they have an impact, but only after considerable delay. One person has studied how long after ‘reversal’ before the medical community accepts the contradiction. John Ioannidis published a paper called Persistence of Contradicted Claims in the Literature. In this paper, Ioannidis looked at claims that had been disproven in the medical literature, he found for one notable example that, “a decade had passed from the contradiction of its effectiveness, [before ] counterarguments were uncommon.” My guess is a decade is about right.
We want to stress the implications of reversal. Reversal implies harm which is multifaceted and enduring.
That squares with a recent report that retractions are linked to patient harm. It doesn’t mean doctors who use evidence that is later shown to be wrong have bad intentions. (In fact, as long as they’re using evidence, they’re ahead of some physicians.) But it does provide yet another reason to read the Archives of Internal Medicine’s “Less Is More” series, which is where this study appeared. It’s some of the most consistently skeptical and evidence-based stuff we see anywhere.
Of course, half of it may be wrong within four years…
Correction, 11:45 p.m. Eastern, 7/11/11: Eagle-eyed Retraction Watch reader Dan Fagin noted that our back-of-the-envelope calculations trying to link this to Sackett’s “half” of all studies being proven wrong within four years were, well, wrong. As he pointed out, we can’t multiply 13 percent x 4 years and get 52 percent, because the universe of studies has also quadrupled over those four years. Apologies for the error, which is solely our fault and not that of the authors of this important Archives paper.