John Ioannidis is perhaps best known for a 2005 paper “Why Most Published Research Findings Are False.” One of the most highly cited researchers in the world, Ioannidis, a professor at Stanford, has built a career in the field of meta-research. Earlier this month, he published a heartfelt and provocative essay in the the Journal of Clinical Epidemiology titled “Evidence-Based Medicine Has Been Hijacked: A Report to David Sackett.” In it, he carries on a conversation begun in 2004 with Sackett, who died last May and was widely considered the father of evidence-based medicine. We asked Ioannidis to expand on his comments in the essay, including why he believes he is a “failure.”
Retraction Watch: You write that as evidence-based medicine “became more influential, it was also hijacked to serve agendas different from what it originally aimed for.” Can you elaborate?
John Ioannidis: As I describe in the paper, “evidence-based medicine” has become a very common term that is misused and abused by eminence-based experts and conflicted stakeholders who want to support their views and their products, without caring much about the integrity, transparency, and unbiasedness of science.
RW: You also write that evidence-based medicine “still remains an unmet goal, worthy to be attained.” Can you explain further?
JI: The commentary that I wrote gives a personal confession perspective on whether evidence-based medicine currently fulfills the wonderful definition that David Sackett came up with: “integrating individual clinical expertise with the best external evidence”. This is a goal that is clearly worthy to be attained, but, in my view, I don’t see that this has happened yet. Each of us may ponder whether the goal has been attained. I suspect that many/most will agree that we still have a lot of work to do.
RW: You describe yourself as a “failure.” What do you mean?
JI: Well, I still know next to nothing, even though I am always struggling to obtain more solid evidence and even though I always want to learn more. If you add what are probably over a thousand rejections (of papers, grant proposals, nominations, and other sorrowful academic paraphernalia) during my career to-date, I think I can qualify for a solid failure. Nevertheless, I still greatly enjoy my work in science and in evidence-based medicine.
RW: You say that your first grant, which you applied for 17 years ago, was “not even rejected.” Tell us about that grant.
JI: It was a randomized controlled trial of antibiotics versus placebo for acute sinusitis. Hundreds of millions of people were treated with antibiotics without good evidence back then, and hundreds of millions of people continue to be treated with antibiotics even nowadays even though most of them would not need antibiotics. I sent in the application to a public funding agency, but have not heard back yet. Probably they felt that requesting funding for a randomized trial and not going to the industry for such funds was a joke. Many public funding agencies are accustomed to funding only research that clearly has no direct relevance to important, real-life questions, so perhaps they didn’t know where to place my application.
RW: You write that clinical evidence is “becoming an industry advertisement tool” and that “much ‘basic’ science [is] becoming an annex to Las Vegas casinos.” Provocative — what do you mean?
JI: Since clinical research that can generate useful clinical evidence has fallen off the radar screen of many/most public funders, it is largely left up to the industry to support it. The sales and marketing departments in most companies are more powerful than their R&D departments. Hence, the design, conduct, reporting, and dissemination of this clinical evidence becomes an advertisement tool. As for “basic” research, as I explain in the paper, the current system favors PIs who make a primary focus of their career how to absorb more money. Success in obtaining (more) funding in a fiercely competitive world is what counts the most. Given that much “basic” research is justifiably unpredictable in terms of its yield, we are encouraging aggressive gamblers. Unfortunately, it is not gambling for getting major, high-risk discoveries (which would have been nice), it is gambling for simply getting more money.
RW: Studying what ails science doesn’t make you popular with other researchers — until they want to publish with you, of course, as you point out in your piece. But those criticisms can also lump you in with those that you describe as “pseudo-scientists and dogmatists…trying to exploit individuals and populations and attack science.” How do you differentiate your own work?
JI: I definitely can’t complain for lack of popularity. I feel privileged to have worked with thousands of other scientists over the years and to have learnt from them. It is not possible to make everybody happy all the time, but the work of my team is aiming to protect science, defend the scientific method, question dogma, and enhance the capability and efficiency of research methodology and research practices. In this regard, it is at the very opposite pole than those who want to attack science, question the scientific method and promote dogmas.
RW: You’re worried that Cochrane Collaboration reviews — the apex of evidence-based medicine — “may cause harm by giving credibility to biased studies of vested interests through otherwise respected systematic reviews.” Why, and what’s the alternative?
JI: A systematic review that combines biased pieces of evidence may unfortunately give another seal of authority to that biased evidence. Systematic reviews may sometimes be most helpful if, instead of focusing on the summary of the evidence, highlight the biases that are involved and what needs to be done to remedy the state-of-the-evidence in the given field. This often requires a bird’s eye view where hundreds and thousands of systematic reviews and meta-analyses are examined, because then the patterns of bias are much easier to discern as they apply across diverse topics in the same or multiple disciplines. Much of the time, the solution is that, instead of waiting to piece together fragments of biased evidence retrospectively after the fact, one needs to act pre-emptively and make sure that the evidence to be produced will be clinically meaningful and unbiased, to the extent possible. Meta-analyses should become primary research, where studies are designed with the explicit anticipation that they are part of an overarching planned cumulative meta-analysis.
RW: What are your hopes for evidence-based medicine moving forward?
JI: The right ideas are there, and there are many superb scientists and clinicians who want to do the right thing, so I am always cautiously hopeful. We should keep trying.
RW: The essay is really personal and full of interesting stories. We’d like to end with a quote from when he was an early career researcher questioning entrenched research attitudes in Europe:
A senior professor of cardiology told a friend of mine that I should not be too outspoken, otherwise Albanian hit men may strangle me in my office. I replied that they should make sure to get correct instructions to my office – turn left when they come up the stairs. I would feel remorse, if the assassins entered the wrong office and strangled the wrong person.
Hat tip: Mohammad Zakaria Pezeshki
Like Retraction Watch? Consider making a tax-deductible contribution to support our work. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. Click here to review our Comments Policy.
This fellow sounds so depressed and defeated… Cheer up, buddy! Your mustache is wonderful
He actually avoids answering the first two questions. From the rest of the interview, it seems that he resents evidence generated by industry, simply because the private sector is not a charity. He wants the evidence to be generated by academia. Evidence is evidence. In fact, industry’s evidence may hold up better to scrutiny, because failure has real consequences for industry, so research findings are vetted more robustly in industry. Whereas a PI in a university can simply say, “Oops, I was wrong… Sorry. Well, according to John Ioannidis, most of our results are wrong anyway, so who cares!”
It would be a nice fantasy if one could trust the private sector. One doesn’t need to look too far to see the influence of profit skewing evidence. Unfortunately, recent history also points to the the lack of objectivity in academia due to the need for funding. Objectivity is always the loser to influences of money, unconscious bias and haste.
Industry can generate fake evidence given enough incentive. Start a trial for your drug that reduces incidence or severity of the platm disease, don’t publish it if it turns out negative. Repeat until you get a positive. 50% chance of success if your drug does nothing.
50 years later and several expensive trials later someone proves your platm trial can’t be replicated, but your water-based medicine wasn’t actually harmful (except to our wallets), and we can’t punish you for doing science and getting unlucky, everyone gets a false positive sometimes. So industry is free to release platm2.
Yeah, industry research has real consequences. Lots of easy money, moneymaking consequences.
I think the answers he gave are directly responsive, what are you talking about? Additionally, I don’t think he is hostile to industry. Actually, he says that industry are the only people willing to fund research with direct relevance to people’s lives, which seems complimentary to industry and critical of publicly funded research.
“Evidence is evidence”? Nope, “evidence” in reality is a construct, you can pose something as an evidence, yet lie about it entirely. Even more so, when you don’t even have to lie, it is enough just to “select” some data, and “publish only” the right results, or to “construct outcome features”, and then pose them as your original goal to report. Of course FDA requires you to post your hypotheses now if I am correct, but this won’t stop anybody from constructing data. His standpoint is clear, he doesn’t blame anybody just describes the system, and he describes it very well.
The most important remedy in my mind is to fund truly independent research (we need many more Mario Negris). Transparency is important but it can’t fix the problem. Nor can trial registry or even meticulous peer reviewers.
Fabulous interview, many thanks Retraction Watch.
More Mario Negris indeed. And I’d plug the MRC’s CTU here too.
I think education is key here. Many statisticians don’t understand how p-values work (http://rsos.royalsocietypublishing.org/content/1/3/140216) and thus can’t truly grasp how shaky much of the evidence-base is, so how do we expect clinicians and students to get it right?
And how on earth do we think we’re going to get good evidence if we leave it up to the industry to produce it? It would be a piece of piss to define what we (patients, medics and purchasers, not shareholders) need to know before adopting a treatment and it is a piece of piss to require that all evidence be produced by independent researchers who are awarded the grant by an independent body through the usual processes.
Two simple steps that solve many of these problems. And all that time that would no longer be wasted arguing with flat earth evidence could be spent teaching our students to become good researchers rather than good at getting industry sponsorship.
We pride ourselves in being up to date with the medical and scientific literature. After reading about this report, what should we believe ? What would be our guide now ? How can we find out before reading an article if it’s credible ?
Funding source has consistently been found to be the single most important predictor of outcome across clinical and health economic studies. Declaring a conflict of interest doesn’t dissolve bias, it red flags it.
Beyond that, we need to teach our students to think critically. I once ran a workshop for a company’s annual jamboree, which had three broad groups, medics, senior scientists and research assistants (many of whom had no degree). They split along status lines for the small group work and it was fascinating watching the process. The paper was a minor head-scratcher, so I called on the medics to feed back first. They presented arguments even more convoluted than those in the paper to try and defend its conclusions. When I turned to the senior scientists, they grinned and said they weren’t gonna go through what the medics had, so I turned to the juniors. They nailed it in a sentence and the room was floored.
We educate people very badly. The reward is in being right, not the quality of the processes we use to arrive at an opinion. So we end up with arrogant bullies who assert their ill-founded opinions most forcefully, instead of the quiet thoughtful ones who are working it out whilst everyone else is jabbering whilst planning to endorse whichever authority is currently in the ascendant.
This is not how we should do science.
PSS – that is a great comment about the teaching of science and your closing paragraph echoes something that I have been saying for some time. Would you be kind enough to post your thoughts on the following piece on my own blog – https://heartontheleft.wordpress.com/2015/12/31/the-coming-year/
I presently working on a piece dealing with the science curriuculum but don’t want to delay getting your thoughts.
Dr. Tony Mitchell
The much more basic highjacking is in the form of what “evidence” means. With very few exceptions, most medicine research dealing with health and illness, has accepted the
reduction of symptoms as “evidence” for acceptable treatments, blessed by such groups
as COCHRANE OR USPSTF . The implicit assumption is that symptom reductions are
valid evidence for inferring health improvement. Researchers are not blind to evidence of adverse effects of a treatment. They openly describe them. They are merely labeled
“side effects”. Sometimes, even death is called a side effect.
Problem is the biology of the human body is not a blank slate, or a set of lego pieces- it
involves sets of very complex,interrelated ,sophisticated systems designed to carry
out certain vital goals. If you ignore what this means, and it is possible that many
symptoms reflect system disruptions, then it is also high probability that you can
apply a treatment to reduce symptoms, and will succeed in both short-term and
long-term adverse effects in the patient. So many diagnoses, GERD , for example
are equating patient complaints of heartburn with GERD. Medications to reduce
stomach acidity are sometimes prescribed without the available tests for Ph levels
in the stomach. Now, with over-the-counter sales, we have more people who
could be suffering major adverse effects because of disruption of vital digestive
systems.
You are wrong, probably because youre not up to date how good evidence measures “success” rates. Look up any meta-analysis, for example statins in your critiziced Cochrane database and you will notice that of course they do measure “effect” or “success” both in symptom reduction but also measures like total death rates, quality of life and so on that include every known and unknown “side effect” – because they simply assess what happens to people one, two five, ten years after they take statins vs who dont take them. If they dont die by cardiov. events but by cancer, bleedings, falls or drowning in a pool because somehow statins have to do something with that – it simply shows in the numbers, not only in descriptions.
Thats very important because youre right that you need to take side effects seriously, not just as mere descriptions, but taking them into account in hard numbers as you do with the symptom reductions.
And so the decision to use of statins or not depends on a number of factors within the patient – judged by the total outcome of the variable you prefer. As a patient and doctor one would surely look for total outcome measures and quality of life, not primary symptom reduction alone. But you can only do this if you know the evidence – so get up to date with it for your patients´sake!
The original point of the 1985 series from McMaster, which is the foundation of EBM, is that outcome measurements should dictate treatment. Symptom reduction is a valid outcome measure, mortality is a valid measure, reduced costs it one as well. But broader spectrum of an antibiotic is not, even if it covered more of the cultured bacteria than the control drug.
Levels of evidence are important only if the difference in outcome is small. If a disease is uniformly fatal, a couple of case reports of cure, are adequate evidence to justify the use of the drug in other patients.
Unfortunately, this simple concept was adulterated.
With respect to medical practice, SYMPTOMS are evidence. There is no black and white line that separates signs and symptoms, and as a physician, you would be ill-advised to ignore what someone reports as a serious “symptom” — alleging it to be merely “subjective,” data to be tossed. What I see as a danger is where “thought leaders” with ties to industry attempt to re-characterize serious symptoms as “subjective” and investigation is thwarted, the patient told they have a “contested illness,” or the dubious “MUS” (medically unexplained symptoms). As a psychiatrist, this practice is extremely dangerous and abusive in the medical context — a critical task being to discern where psychiatric “symptoms” reported by the patient represents an undiagnosed medical illness not comprehended by the internist, pediatrician, rheumatologist, etc. But we have physicians with virtually NO TRAINING in psychiatry (except a 6-week rotation as medical student) who are either dumb as rocks, or with commercial incentives proclaiming syndromes are subjective and psychiatric in nature! Conduct that is nothing more than outrageous w/physicians acting outside expertise/practice boundaries. However, as you say above, then there are examples like heartburn being equated with GERD, and no such conclusion like this can be drawn – you are right. Physicians are not taught to think critically. Medical education needs serious overhaul. Clinical reasoning is given short stick in medicine where memorization is valued, and ability to reason based on clinical presentation is not prized. Legal education? the opposite — where legal reasoning is more highly valued than necessarily immediately arriving at “right answer.” With regards to science and academic medicine, to what degree has the Bayh-Dole Act (1980) changed the culture, where decisions about what work to underwrite by PHS is based on commercial possibilities — and basic science is neglected because no immediate commercial application to be patented or sold to biotech/pharma? Legislators knew risks associated with Bayh-Dole passage (essentially marrying academic science and medicine to industry), but chose to ignore. Are present circumstances the unintended but anticipated abuses requiring serious thought be given to amend Bayh-Dole — but U.S. Congress unlikely to intercede, investigate, or pass legislation because campaigns are underwritten by Pharma? Hard to cabin abuses now that they have become routine. Every university, researcher, and even regulatory agency will be out there protesting, worried cuts will be made to change corrosive incentives. Scientists who have not bought into this regime bullied. What happened to academic liberties? How many scientists on choke chains to depts to produce brilliant patents w/public tax dollars that have underwritten the research — but research no longer directed towards public interest? Brilliant commentary by Dr. John Ioannidis. Thanks to Retractionwatch coverage!
There is no black and white lines between symptoms and signs? Yeah… maybe.
But when you have loads of medical records full of stuff like “mother said child has symptom X” and just next to it there is the mention “no clinical sign”, you should start wondering at what point everything is happening in the psyche of someone who is not the patient.
Symptoms and signs are different beasts. No clear line between them, but a world of difference nevertheless. Confusing them can kill. Even in psychiatry.
Wonderful interview! These types of stories strengthen the value of RW.
As a practicing cardiologist and scientist, I find this discussion and comments very thought provoking. First, “evidence-based medicine” is not an exact science and it is constantly changing as we learn more. Second, evidence-based medicine has been hijacked! Medicine and healthcare is a noble profession that has been hijacked and misused by government bureaucracies, big business interests, special interest groups and healthcare system bureaucrats who are no longer responsive to frontline healthcare providers and patients. The effective practice of medicine must be lead by physicians and other providers who use their education, background in science, and their experience in the art of medicine to provide optimal care for each individual patient. We are now required to live by many poorly crafted “evidence-based medicine” rules, regulations, regulation, data gathering and reporting requirements that are overwhelming our uniquely American healthcare system. The vast majority of these requirements detract, rather than assist us in the practice of medicine, because they are promulgated by unresponsive bureaucrats, not providers of frontline healthcare. We’ve become slaves to poorly designed electronic health records and health information technologies that have markedly decreased our ability to practice medicine efficiently and cost-effectively. Most of the time we spend in patient visits is now often typing on a computer to document mandated requirements and not interacting with our patients. Things are beginning to change — much of our electorate is very angry and clamoring for change. We need to hold our elected officials and other unelected bureaucrats responsible for failing to serve us. It’s time for major change — see American Healthcare Reform: Fixing the Real Problems — for a quick overview of the real issues. Change is coming and we all need to be responsible for moving healthcare forward like it should be practiced. Twitter @EarlWFerguson
Very interesting comment. Thanks for posting it.
Prof. Ioannidis, you are truly inspiring and a genius. Ever since you told me about the vast array of computational methods, data-sets, analysis etc. I’ve been thinking about it and wrapping my head around these new developments. Holding two decades strong against the corruption, and unconscionable things that muddy medical and science research? Sir, you are a patriot, and you continue to inspire me and countless others. They talk about you in the trenches synonymous with a war hero. If there’s any hope for initiatives like “precision medicine” I do hope clinicians and scientists train themselves in meta- techniques, and be courageous enough to not bow out to industries.
I share John Ioannidis’ concerns that Cochrane reviews “may cause harm by giving credibility to biased studies of vested interests through otherwise respected systematic reviews.” There are, for example, individual Cochrane reviews on homeopathy which suggest there is reasonable although limited evidence of efficacy. He suggests that “systematic reviews may sometimes be most helpful if, instead of focusing on the summary of the evidence, highlight the biases that are involved and what needs to be done to remedy the state-of-the-evidence in the given field”
Could Cochrane reviews redeem themselves if they routinely included a “Cochrane protocol” for primary research to answer the review question and slot straight back into the review, – a step towards remedying the poor state of the evidence for that question? That would be a unique and constructive innovation. Empty reviews or those containing potentially biased studies could become positive springboards for new well-designed studies to answer important questions.
As John says, “Meta-analyses should become primary research, where studies are designed with the explicit anticipation that they are part of an overarching planned cumulative meta-analysis”
First, I need to state that I am neither a researcher or even officially by academic standards a scientist since I don’t have my doctoral degree. If you only believe that researchers or scientist should be qualified to comment, you can stop reading now. I am currently an Employee Assistance Professional and therefore by necessity a Behaviorist (note: I believe actions speak louder than words). But I have worked within a large research institution with an affiliated academic medical center for almost 3 decades and during that time have written and applied for grants, participated in drafting articles for peer reviewed journals, reviewed submitted documents for their validity, submitted and obtained process approval to the IRB, written policy and made other determinations for coverage of evidence based procedures within a health plan and worked to support clinicians and faculty (and the support staff that actually do all the work within the lab or research project) so I have a fairly broad understanding of evidence and the politics of interpretation of the scientific process from birth of the idea to approved bedside care.
The “evidence” that our medical scientists produce is directly relevant to funding sources, nothing more and nothing less. As an example, let’s glance at obesity. Obesity is an acknowledged problem, but the number of relevant studies on how to help people successfully modify their diet and activity level to control their weight is horribly lacking. Why, because normal funding sources are not interested. Since pills only seem to cause side effects and law suits, pharma has lost interest but are still working on it. The government has ceased funding most pure science on public health issues. Commercial entities don’t do research, they advertise and promote and consumers purchase or not as they choose. If they find something of value, they don’t share with the competition. The main body of evidence on obesity control has happened only since the popularity of bariatric surgery which promotes surgery as a potential cure for diabetes, notwithstanding the complications and failure rates with the surgery since those clients get excluded from the results. Since funders get to review the proposed research methods and specify the process and exclusion criteria prior to granting funds, the results will be stacked in the funder’s favor. End of discussion. Review groups like Cochrane can only review published research, so if there is nothing but biased research published, the meta-analysis will be biased.
I agree with every point that is made by John Loannidis and independently have come to the same conclusions about evidence based care. He is describing the actions and not buying into the hype and as I stated earlier, actions speak louder than words. The true irony for me is that from my office window, I can observe an exit ramp to a local freeway and I see people dressed in jeans, old hiking boots, flannel shirts and a ball cap holding cardboard signs asking for money. These people are either invisible or pitied depending on the driver’s point of view. When I walk across campus, I see people dressed in jeans, old hiking boots, flannel shirts and a ball cap discussing how they just were awarded $750,000 a year for the next three years to study evidence based care. Both individuals are relevant to society but one is revered, may become world renowned and one is not and won’t be known by anyone outside of their immediate circle. I value them both and I am not sure if the one with the card board is the lesser because at least they are not wasting large sums of money to make a pre-determined point.
I understand Dean’s point of view, however, do not agree that “evidence is evidence”. Many of my clients (I am a pharmacist and pharmaceutical copywriter) develop their own evidence base leveraging CME, scientific publishing, DTC advertising and physician marketing in order to achieve their business objectives. In some cases, these efforts have led to catastrophic outcomes such as the current opioid epidemic, the heavy use of atypical antipsychotics in children or the overuse of erythropoietin in dialysis patients. Although no drug company is going to say this, their primary goal is not appropriate use, it’s to increase market share and sales volume. John’s stand is so important – what we need is a national conversation on the subject and for academia to take a larger role in physician/consumer medical education.
“I am not bound to win, but I am bound to be true. I am not bound to succeed, but I am bound to live by the light that I have. I must stand with anybody that stands right, and stand with him while he is right, and part with him when he goes wrong.” Abraham Lincoln