John Ioannidis, a professor at Stanford University and one of the most highly cited researchers in the world, has come up with some startling figures about meta-analyses. His new paper, published today in Milbank Quarterly (accompanied by this commentary), suggests that the number of systematic reviews and meta-analyses in literature have each increased by more than 2500% since 1991. We asked Ioannidis — who is perhaps most well known for his 2005 paper “Why Most Published Research Findings Are False” (and was featured in a previous Retraction Watch Q&A article) — why such a massive boost these publication types in scholarly literature is potentially harmful.
Retraction Watch: You say that the numbers of systematic reviews and meta-analyses have reached “epidemic proportions,” and that there is currently a “massive production of unnecessary, misleading, and conflicted systematic reviews and meta-analyses.” Indeed, you note the number of each has risen more than 2500% since 1991, often with more than 20 meta-analyses on the same topic. Why the massive increase, and why is it a problem?
John Ioannidis: The increase is a consequence of the higher prestige that systematic reviews and meta-analyses have acquired over the years, since they are (justifiably) considered to represent the highest level of evidence. Many scientists now want to do them, leading journals want to publish them, and sponsors and other conflicted stakeholders want to exploit them to promote their products, beliefs, and agendas. Systematic reviews and meta-analyses that are carefully done and that are done by players who do not have conflicts and pre-determined agendas are not a problem, quite the opposite. The problem is that most of them are not carefully done and/or are done with pre-determined agendas on what to find and report.
RW: According to your paper, these types of papers have become “easily produced publishable units or marketing tools.” What do you mean by that? How should systematic reviews and meta-analyses be used?
JI: In the past, a company that wanted to promote its products had to get a number of opinion makers — i.e. prestigious academics — to give talks and write editorials and other expert pieces about these products. As expert-based medicine of this sort declined and randomized trials became more influential, a company shifted its preference to trying to manipulate randomized trial results so as to convince the community. Now that systematic reviews and meta-analyses have become even more highly recognized than randomized trials, emphasis is shifting on dominating the results and conclusions of systematic reviews and meta-analyses. So, this has become the latest marketing tool, still serving expert-based medicine in essence. Moreover, given that the methods of performing systematic reviews have become more widespread and easier to apply (actually mis-apply, most of the time) lots of authors see systematic reviews and meta-analyses as a way to build a CV.
RW: Are there better ways to present systematic reviews? What about as living documents on the internet to which new results can be added, as a commentary accompanying your new paper proposes?
JI: In principle any new study should start from a systematic review of what we already know (to even justify the need for its conduct and its design) and should end with an updating of what we know after this new study, again in the systematic review framework. So, I am in favor of this concept. At the same time, I feel that we have missed a great opportunity, because we keep thinking about systematic reviews and meta-analyses as retrospective assessments of past evidence. We should think of them more as prospective integration of evidence for larger research agendas that are prospectively designed. They should become the main type of primary research, rather than just try to compile fragments of published, distorted, selectively available information. Fields that have promoted consortia and multi-team collaborative research are well familiar with this approach.
RW: What can universities, journal publishers and individual academics do to tackle “unnecessary, misleading, and/or conflicted” reports?
JI: Like in any type of research, it is an issue of defending quality and high standards. We see problems of low quality and low standards in almost any type of research, so systematic reviews are not an exception. The reward and incentives system can play a key role in encouraging or discouraging some types of scientific behavior, and universities as well as journals are key gatekeepers. Individual academics can also aim to get better training in proper methods for conducting meta-analyses, and set higher goals in the systematic reviews and meta-analyses that they conduct.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
Clearly, there has been a huge proliferation of meta-analyses in recent years. There has also been a proliferation of gene manipulation papers, because the CRISPR-Cas9 methodology made it possible to target genes for insertion or replacement. New tools beget new ideas, and some new ideas become fashionable. The fact that there are more meta-analyses now than in the recent past simply means that meta-analytic methodology is useful.
How can the reader decide which meta-analyses are “unnecessary and misleading” and which aggregate valuable findings? That is an exceedingly difficult challenge and no guidance is given. But it is a grave error to imagine that any “player” is free of conflict. When a scientist invests months or years of time in a project, that scientist cannot be considered unbiased, even if they aren’t paid by a corporate entity. To imagine that academic scientists are inherently more objective than other scientists is naïve and ignores the fact that the vast majority of retractions come from academic scientists.
Here are a few proposed guidelines, all of which attempt to hold a meta-analyses to at least the level of quality of the papers which they are reviewing:
1) At a minimum, the authors of the meta-analyses should not include any of their own papers in the meta-analyses.
2) Any analyses or ratings produced should be algorithmic and transparent, with all underlying computations and assessments published, ideally proactively publishing the proposed analyses, as would be done for a clinical trial.
3) Any paper that wants to be called a Meta-Analyses must be peer reviewed.
4) Any review that wants to avoid the theoretically higher level of rigor required for a Meta-Analyses must clearly state that it is an editorial or speculative opinion, not a formal analyses of any kind.
5) Meta-Analyses should exclude both unpublished papers and papers published without peer review.
From my experience many systematic reviews are not correct for the simple reason that they
magnify the errors and falsehoods of the underlying studies or just plain confuse themselves
with over analysis.
From Stephen Senn “Meta-analyst. One who thinks that if manure is piled high enough it will smell like roses.” This is the opinion of a medical statistician where usually the studies are of a better quality than psychology, for example.
I note that this is specifically focused on medicine.
I know that for myself, while meta-analyses have grown in popularity in experimental psychology and neuroscience, I haven’t seen the problem described here. On the contrary, I’ve seen nothing but benefits. Quite often in the past a topic has had a single person interested in the meta-analysis. The increase has allowed for a couple more to provide balance or confirmation.
What about the bias in meta analyses due to publication bias.
In the era of innovation rush there is probably a publication bias related to the fact that a study confirming already published observations is less prone to be accepted (for lack of novelty reasons) than a study which contradicts the first study published. To avoid this, researchers usually adopt a different study design or result presentation to claim a bit of novelty. In meta analyses this results in a lack of homogeneity in study procedures and designs and probably in statistical power of the final conclusion.
I am rather surprised by this article. A fundamental principle of research and of SRs in particular, is critical appraisal. This process itself needs to be based on evidence. I wondered why this article does little to help the reader. For instance, protocol-driven reviews as produced by the Cochrane Collaboration are, I believe, state of the art. I use this phrase carefully as i don’t believe SR production to be entirely scientific. But I do think Cochrane is as good as it gets.
Not only are the numbers of systematic reviews and meta analyses reaching “epidemic proportions” but many are flawed or biased because the researchers have used the wrong databases or their strategies for searching for evidence are inadequate. As a result relevant evidence has been missed.The number of systematic reviews where medical databases only have been used where the subject is social research or policy or social care are legion!
I would add that it is necessary to promote in a stronger way the share of the data of research, but also the data of how systematic reviews and meta-analysis are conducted; in the last case, the data should be publicly available in order to allow an easier update without start all over again and in order that anyone can verify the study and find mistakes.
Systematic reviews are time consuming and do require intellectual input but do not require the hard foots logging graft that real research does, leaving aside the hours of justification to largely negative ethics committees. Hence the trawl through trials of dubious rigorous in areas of ethical softness that then get included In meta analysis of equal softness. It would s much easier to pick your way through the efforts of others to produce data than to collect your own. And the participants. Un thanked pawns in the academic game of publish or perish,
It should also be mentioned that another catalyst for the increase in meta-analyses is surely that it is much easier to gather and store multiple studies because of the internet and modern computing power.
Major problem is that many of those who perform systematic review has no really good background of methods. Few days while helping a person who had several systematic reviews of high impact factor, the person told me had no idea of what confidence intervals are. So how a person who has no idea about CI, or bias etc can be able to perform a systematic review. Nowadays people follow the ckeck list and that is it. A case-control study is a case control no matter how controls were collected. A clinical trial is considered the best no matter if time varying variables were correctly evaluated in a statistics or not and etc. So people are just putting together a bunch of bad studies and concluding over that.
People are not interested in really learning statististics and methods. Moreover, PhD nowadays need to finish their PhD with 10 or more papers to get a job, so there is no time to learn. I am doing some mixed models and doing residual analysis now. Somebody else had done the statistics before , a very prolific reserarcher, but residuals are a mess… so. However that researcher is publishing a lot but the researcher had no idea what residuals are for. Of course he/she can publish a lot. Of course students think the prolific researcher is the best one and no need to really learning stuff….
There are many different problems a meta-analysis can have. Thinking back to some I’ve looked at:
Bjelakovic et al. 2007, “Mortality in randomized trials of antioxidant supplements for primary and secondary prevention: Systematic review and meta-analysis”, Journal of the American Medical Association, Feb. 28 2007. This study concluded that vitamins A and E are toxic. The problem was that it combined data from studies which used widely varying dosages, including several which used mega-dosages known to be toxic. Sure, many things are toxic at very high doses. But the study used linear regression to compute toxicity, which assumes the dose-response curve is linear across the entire range studied–which is never the case for any medicine, and especially not when your dosage ranges over 2 orders of magnitude.
Magnuson et al. 2007. Aspartame: A Safety Evaluation Based on Current Use Levels, Regulations, and Toxicological and Epidemiological Studies. Critical Reviews in Toxicology 37:629-727. This is a classic bad industry-funded study. (I DO NOT BELIEVE THAT ASPARTAME CAUSES CANCER, but I do believe attempts to determine that have been invalidated by industry intervention.) The paper has a list of 10 authors, and for each author, lists a university affiliation. At least 9 of these authors had taken money from firms that sold Aspartame or otherwise had a commercial interest in the paper’s conclusions, and the paper was funded by Monsanto, but none of this was not disclosed. (The fact that Gary Marsh worked for the Formaldehyde Institute is almost certainly why the paper had a section arguing that, contrary to what everyone else believes, formaldehyde is non-toxic, and therefore Aspartame cannot produce formaldehyde toxicity since there is no such thing.) When the most-often-cited meta-review on a subject is discovered to have been funded by a financially interested party, the publication should print a notification of that fact. The paper spent 1328 words critiquing studies that found Aspartame toxic, and only 222 words critiquing studies that did not.
Some time ago, I saw an informational graphic about the alleged link between vaccines and autism. It said, if I recall correctly, that out of 60 studies on the matter, not one had indicated a link. Now, I DO NOT BELIEVE THERE IS A LINK BETWEEN VACCINES AND AUTISM. However, what are the odds that 60 out of 60 studies would fail to find a link between vaccines and autism at 95% confidence? .95 ^ 60 = .046. This would prove, with 95% confidence, that studies in the literature are biased against finding a link between vaccines and autism. (If a hypothesis is false, a meta-review of studies which try to reject a hypothesis with 95% confidence should find that 5% of them accept it.) I was curious whether this was publication bias or meta-review bias, so I looked at 6 meta-reviews (see http://lesswrong.com/lw/ki4/too_good_to_be_true/) and found that studies finding no link had a 2.8% chance of being rejected for consideration by a meta-review, while studies finding a link had a 75% chance of being rejected.
A more blatant example of a biased “meta-review” is the 2006 IDSA guidelines on Lyme disease, which simply chose not to cite any studies (the majority of everything published on the topic since the 1980s) which contradicted its pre-ordained conclusions.
But the biggest problem with meta-reviews is that they aggravate a problem individual studies already have: They assume homogeneity of subjects. This is stupid. We know this is stupid. We should stop doing it. Every medical journal article ever published that says “X is safe” should instead say, “We have rejected the hypothesis that X causes a single consistent toxic reaction in every human being with 95% confidence,” because that’s all they’ve really done. Everything we know about breast cancer risk factor would have been rejected by international meta-reviews, because the risk factors vary from ethnicity to ethnicity, culture to culture, and decade to decade.
In reference to the statement, ” This would prove, with 95% confidence, that studies in the literature are biased against finding a link between vaccines and autism. ” I believe that Phil has made an analytical error. He indicates that the fact that 60 out of 60 studies have negative results shows bias at the 95% confidence level. But what if the confidence level of the result (60 out of 60) is vastly higher than 95%? So high that it would take at least 61 or more studies, with only one being positive, to get anything other than negative results? Then there is no confidence of bias, just an extremely high confidence level that there is no link. The same applies to his finding that 3 out of 4 positive studies were rejected on methodological grounds… this is not necessarily an indication of bias, but more likely an indication of poor methodology (or bias) in the 3 rejected studies.
Thus, the statement that the result (no link) is shown at the 95% confidence level is simply insufficient; the actual confidence level is so high that none of the studies conducted showed a link. At least, that is my reasoning, based on a Bayesian analysis: the prior probability of a lack of a relationship between vaccines and autism is so great that the probability of a study that is properly conducted, with good methodology, coming out with a 95% confidence showing of a link, is vanishingly small. Thus, the fact that a study shows a “95% confidence” of a link between autism and vaccines makes it more likely that the study is methodologically unsound than that it really shows 95% confidence.
Take the physics analogy: dropping an apple off a tower will virtually always result in the apple falling to the ground. In occasional case where the apple does not reach the ground, we can guess that a passing bird has snared the apple, rather than the alternative, namely that gravity does not operate with 99.999…% confidence.
On the other hand, with reference to aspartame and formaldehyde toxicity, we “know” that exposure to formaldehyde may produce toxicity or even cancer. (See National Cancer Institute papers on formaldehyde: they classify it as a “probable” human carcinogen, partly based on rat studies, and describe known toxicity from high-level exposure. https://www.cancer.gov/about-cancer/causes-prevention/risk/substances/formaldehyde/formaldehyde-fact-sheet) Therefore, properly conducted studies of aspartame might show that high levels of exposure to aspartame cause toxicity, based on the fact that “known” metabolic breakdown products of aspartame include formaldehyde. (I didn’t look for references on metabolic products of aspartame, but I’ll guess that most biochemists will vouch for it.) Such studies should at least occasionally show this: Bayesian analysis leads us to expect that aspartame might cause “significant” toxicity. Therefore, with industry funding, the probability that the review that completely clears aspartame is biased is significant.
i have learnt in my epidemiology classes how meta-analysis and systematic reviews helped in developing standards and policy making in public health. These days i notice that meta-analysis has become like writing a review – go through few papers and then do the analysis with no statistical power….
Speak of the devil!
https://www.medrxiv.org/content/10.1101/2020.05.13.20101253v1.article-metrics
My major concern and impression being an epidemiologist (considering I had a good background with my PhD at UofM), and working with clinicians who are not epidemiologists, is that people decides to make systematic reviews and meta-analysis because it is fashion. Usually they don’t have ground knowledge of methods, science or statistics. The old problem of reviews in the past continues the same: researchers’ lack of knowledge of science and methods. It so that in recent years, people from systematic reviews came out with “risk bias” because no one was paying attention in the internal validity of studies. Now they fill out the “risk bias table” without knowing what selection or information bias is. A friend of mine has several first author systematic reviews in excellent journals with a lot of citations and asked months ago what standard error and 95% IC was. Many studies that are used in meta-analysis have so poor desing with clear selection bias and authors and readers have no idea. The worst: recently a famous doctor told me he doesnt have time to read papers, so he only read systematic reviews. So what to expect of many of our future researchers?