Editor’s note: This post responds to a Feb. 13 article in The Atlantic, “The Scientific Literature Can’t Save Us Now,” written by Retraction Watch cofounders Adam Marcus and Ivan Oransky.
The contentious issue of what — and more importantly who — to believe, when it comes to medical science, is at a critical moment. Watchdog organizations such as Retraction Watch provide a great service to science and the public, by exposing junk scientists and their products, helping to disinfect the field with their sunlight. I commend Mr. Marcus and Dr. Oransky for their sustained efforts in this meta-discipline.
However, policing the scientific literature is a tricky business. In particular, one must be careful to apply the same standards one demands of others to one’s own work. Agreeable as many of their points are, Marcus and Oransky’s article discrediting Mawson and Jacob’s study (which Robert F. Kennedy Jr. cited during his confirmation hearings) falls woefully short of meeting even basic scientific editorial standards. This failure imbues their article with the same yellow hue that they decry in others’ journalism.
Resorting to a combination of circumstantial evidence and ad hominem characterizations, Marcus and Oransky insinuate that the study, which purports to show a link between current childhood vaccinations and neurodevelopmental disorders (NDDs), is scientifically flawed and thus not to be trusted. Yet nowhere do they provide any substantive critique of the study itself. Indeed, it is unclear from their article whether Marcus and Oransky even read the study they seek to discredit.
Rather, their argument runs as follows: The study’s first author is no longer an academic. The journal that published it was not indexed by an organization with scientific credibility. Several editorial board members seem shady. The journal can’t even be bothered to spell an editor’s name correctly. And, by the way, the hypothesis Mawson and Jacob investigate has already been “thoroughly debunked” and is therefore pointless to question.
Really? So henceforth, no investigator shalt ever study whether vaccines are associated with, not just autism, but the spectrum of NDDs (as Mawson and Jacob do)? Which vaccines are Marcus and Oransky specifically referring to? All vaccines? Or only the MMR? As they must know, it is nigh impossible to prove scientifically that a hypothesized association doesn’t exist, let alone for all vaccines, and a roster of disorders. Would they extend this inference then to every new vaccine that enters the market? What if someone’s findings don’t align with what is already “known”? Should they be tossed out? Buried?
It is not clear what relevance Mawson’s employment status has to the validity of his and his co-author’s study. Many scientists work at private research institutes, or for corporations, outside of academia. That two editorial board members have, according to the claim, tainted scientific reputations, is circumstantial. The lower scientific stature of Science, Public Health Policy & the Law might seem damning, were it not for the fact that, as Marcus and Oransky’s own work has shown, publication in a venerated journal is no guarantee of a study’s reproducibility, or even veracity. Conversely, publication in low-tier journals may be the terminal fate not only of shoddy research, but unpopular ideas. The former deserves no defense; the latter does.
Worse, Marcus and Oransky sandbag their own article with unsubstantiated, dubious claims, meant to advance their argument but ultimately leaving it naively exposed to Hitchens’s razor (“What can be asserted without evidence can also be dismissed without evidence”). How could they possibly know that “far more papers should be retracted than are retracted”? Not every publication with errors discovered post hoc need beget a retraction. Scientific journals commonly publish errata. Further, on what grounds can they plausibly claim that “most” published papers “serve no purpose whatsoever”? I allege they do not have exclusive insight into what purpose a given paper serves, especially considering posterity. Short-run citations are not the same as purpose.
On my read of Mawson and Jacob’s paper, it appears to be a straightforward analysis of 47,155 Florida Medicaid claims records from 1999 to 2011. Nothing immediately screams “junk.” The authors use methods common for other claims-based case-control studies, and control for confounding effects of age, gender, and congenital anomalies in their analysis. Their paper describes a relatively balanced assessment of the study’s strengths and weaknesses.
Since the study was rejected without review from other journals, it is unclear what the basis for rejection was, but this editorial action is common and not necessarily an indication of study quality. Publication bias, which refers to how a study’s results affect its publication status, could easily be a factor influencing preceding editorial decisions. In any case, the records are publicly available, and the methods appear duly described, enough to make the study reproducible, should anyone want to analyze it independently.
My prior research in vaccine safety, infectious diseases and neurologic disorders, treatment of vaccine-preventable cancers, and quality of medical scientific literature have made me no champion of the vaccine-autism theory. As for disclosures, I have consulting agreements and research funding from pharmaceutical companies, including Johnson & Johnson and Merck, makers of vaccines. I love vaccines, but I will defend scientists who want to study their safety, and report what they find, where they are able to find a receptive outlet, irrespective of how well their findings comply with other studies.
We all should defend studies against sloppy attempts to discredit them. Because in the current tumult of distrust in experts, it has never been more crucial that Marcus and Oransky, and influential media that provide them a platform, retain their scientific high ground. Status and slick innuendo do not confer the right to skirt rigorous discourse when it comes to critiquing science. Weak arguments from strong voices also do our field a disservice. “Settled science” claims are lazy and readily backfire when thrust upon a distrusting public. They also betray a humanistic disconnect, which I suspect lies at the root of this distrust.
Every month I see patients who refuse to consent for the best known cancer treatment, despite my recommendation, for various personal reasons. Often these reasons are irrational. Many are rooted in misunderstandings courtesy of junk science. My job isn’t to castigate patients for their irrationality. It is first to establish their goals of care. Within that, I can convey my interpretation of the present state of the pertinent medical science. Then, it’s up to them. Among my most important ethical obligations, described in the Declaration of Geneva, is to respect patients’ autonomy, even if it conflicts with what I believe to be in their best interest. Policing junk science can slow, but won’t eliminate irrational beliefs. Nor will policies that threaten individual autonomy, such as making vaccines coerced or compulsory. This merely fuels distrust.
With respect to vaccines and autism, based on work from DeStefano et al., Jain et al., and others, I would conclude that the balance of scientific evidence to date does not support the hypothesized association. This can be stated without having to malign studies that cut against the grain. Yet such mouthfuls are insufficiently sensational, and do not lend themselves to pithy soundbites in mainstream media — which is the point here. Scientific discourse and mainstream journalism operate by different standards. The conventional forum to raise concerns about an article is within the journal’s pages, as a Letter to the Editor, typically alongside a rebuttal from the author(s). Since The Atlantic has chosen to dedicate its pages to scientific discourse, I trust it will provide space for rigorous dissent.
Loren K. Mell, M.D., is a physician-scientist and clinical trialist at the University of California San Diego specializing in head and neck cancer. He previously studied vaccine safety at the Center for Health Studies in Seattle and his current research areas include treatment of vaccine-preventable malignancies and quality of scientific literature.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
I agree. We should, every second of every day of our lives, entertain every crackpot hypothesis that anyone ever brings up because to do otherwise is unscientific. Expect my next paper on whether the sun will rise tomorrow, and if you don’t spend at least two hours reading it, you hate science
Agreed.
Brandolini’s law (bullshit asymmetry principle):
The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.
But I respect RW’s willingness to publish this.
Thank you for publishing this critique of your article. It is important to know that Retraction Watch is willing to subject itself to criticism. Dr. Mell makes some valid points that I hope you will take seriously.
One important point is that journals reject articles without review for many reasons, most of which have nothing to do with quality of the paper. Often they won’t even have looked at the paper closely enough to judge quality – it’s often a matter of not finding the topic of interest to them, especially in high impact journals with many more submissions than they can publish or for papers with null/negative results (the file drawer problem).