I agree with your conclusions completely, and your paper is still terrible.

James Heathers

Yesterday, dozens of scientists petitioned the Proceedings of the National Academy of Sciences to “retract a paper on the effectiveness of masks, saying the study has ‘egregious errors’ and contains numerous ‘verifiably false’ statements,” as The New York Times reported. One of those scientists was James Heathers, whose name will likely be familiar to Retraction Watch readers because of his work as a scientific sleuth. We asked him to share his thoughts on why he signed the letter.

Recently, The Lancet retracted a paper critical of the drug hydroxychloroquine in the treatment of COVID. The retraction was requested by three of the paper’s authors, who had requested a copy of the underlying data from the fourth. The data never came. No data, no paper.

The above took a few weeks. To many people, it must have looked like silence. But the entire time, out of the public eye, a furious and detailed global discussion between scientists, statisticians, and epidemiologists about the accuracy of this paper was boiling. Frankly, almost no one believed this paper. The data were too regular. The access that was reported to hospital databases was too unusual. The amount of work done didn’t fit the parameters reported.

It was … “off.” It was obvious. Something was amiss.

I spent two days, two days I didn’t have, trying to reconstruct elements of the dataset from the results presented in the paper, trying to demonstrate statistically that the data was bogus. I made frustratingly little solid progress, and then the work was retracted anyway. Such is life in the data thug business.

But I keep returning to one element of the narrative that people often miss: the strong majority of the scientists who were critical of this paper believed in the reported conclusions. This paper strongly asserted that hydroxychloroquine was largely ineffective as a COVID treatment, and potentially dangerous (which as of Monday is also the official position of the FDA) ).

I was worried about this months ago , and this paper asserted that I was essentially correct. But what do you do when a paper you can’t trust agrees with you completely?

The answer is: the same as any other situation – immediately spend some time trying to interrogate it by any means necessary.

Yesterday, several other scientists began a similar discussion. 44 scientists wrote to the journal Proceedings of the National Academy of Sciences yesterday to request another retraction. I’m one of them. I usually loathe petitioning anything empirical, and the co-founders of Retraction Watch agree that calls for retractions aren’t usually a good idea. I don’t like the idea that we need to bolster the best arguments with the most visible social support. And yet, I signed up for this one without hesitation.

As before, the paper — published just last week — touches on an important topic: (the relative role of the use of masks compared to other COVID mitigation politics). Again, it is overwhelmingly likely that all of these critical authors agree with large parts of the study’s conclusions (that masks are relatively effective at limiting person-to-person viral transmission). And, of course, we feel the paper is untrustworthy (in this case, inaccurate, superficial, and methodologically unsound).

Retraction Watch readers will not need this explained, but when this happens, there is a large segment of the population who wonder what the hell we’re doing. From a public relations perspective, this is nonsensical – criticism of high profile conclusions that bolster your own arguments? Why not let sleeping dogs lie? Why not admit papers into the scientific literature which are convenient but untrustworthy?

There are four primary reasons. First, methodologically weak science is often easier to produce, and when a lower standard of evidence is accepted once, it is much easier to justify in the future. As such, one weak high profile paper forms an invitation to other scientists to produce their own. Left uninterrogated, bad methods can metastasize like tumours.

Second, The Ruiners never sleep. Scientists working in the public interest are not the only people who are capable of levelling criticisms. Imagine some clever and motivated party who is dead against the public policies on mask usage — this is not hard, they exist. It is a convenient argument to many people at present. Well, these people are entirely prepared to level the same arguments. While the strong bulk of the available evidence would not support their position, and they will not change consensus, their criticisms in this case will be strong and valid. The following extended conclusions — arguments for mask wearing are weak in general! scientists are liars!” will not be. But they will at least have the credence afforded by being right in this case.

Third, it will be a lot worse for public perception if a disagreement like this is left alone. Retraction Watch is full of these stories, the toxic and obviously fraudulent paper which took two years to be retracted. We do not have two years. Often, we do not have two days. During a pandemic, high-profile scientific papers change both public policy and private actions astonishingly quickly. Withdrawing a paper is much simpler than withdrawing government-mandated changes to human behavior.

Finally, we get to the most obvious reason — that the public, whose money supports the vast majority of the formal scientific enterprise, deserves the most accurate science possible regardless of the circumstances. Do we risk muddying the waters in the meantime? Yes. Will this be frustrating for many people who are generally unfamiliar with the scientific process? Also yes.

The importance of making sure evidence is ironclad at present is not some abstract game that will be played out over years. Science has become a much faster conduit to real life, its immediate importance magnified many times over by the pressing need for information. So much virus, so little time.

Scientific criticism may well be misinterpreted as a lack of consensus. It may well reflect badly on the scientific process. But, at present, it should also be considered mandatory by any scientist with a conscience. Quick and decisive responses to unreliable research are not optional at present, they are mandatory. If you disagree, I recommend an alternative career. Maybe one where being an ineffectual courtier isn’t danger to the planet.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at team@retractionwatch.com.

7 thoughts on “I agree with your conclusions completely, and your paper is still terrible.”

  1. “Frankly, almost no one believed this paper. The data were too regular. The access that was reported to hospital databases was too unusual. The amount of work done didn’t fit the parameters reported.

    It was … “off.” It was obvious. Something was amiss.”

    So how the paper got approved by peer preview? shouldn’t the peers be also on the spot?

    “Scientific criticism may well be misinterpreted as a lack of consensus. It may well reflect badly on the scientific process. But, at present, it should also be considered mandatory by any scientist with a conscience. ”

    “At present”? That is a very damaging paragraph, it is almost Orwellian, i guess it is how the language control science today.
    Seems you are forbidden or don’t want to say that lack of consensus is often part of scientific criticism.
    Consensus turned into religious dogma for science.

    Fits, in Revolutionary periods it is not possible to say No. Popular Front of Scientists…

  2. From PNAS website:

    What happens to a submitted paper, and who makes the final decision?

    The Editorial Board member (who is also an NAS member) receives initial submissions and is responsible for the final decision. The Editorial Board member can reject the paper at initial submission, elect to serve as the NAS member editor, or select another NAS member editor. Currently, there are more than 210 Editorial Board members.

    Next, the NAS member editor can reject the paper or send the paper for review. After review, the member editor makes a recommendation about acceptance to the Editorial Board member. Multiple rounds of review and revision are strongly discouraged. All 2,800 NAS members are eligible to serve as member editors; 1,236 member editors handled papers in 2018.

    In rare cases where there are no NAS member editors available to review a submission or expert in the subject matter, a nonmember guest editor may be used. The final decision is still made by the Editorial Board, and the name of the Board member appears on the paper along with the name of the guest editor.

    For additional information on the PNAS review process, please visit Editorial and Journal Policies.

  3. Why not admit papers into the scientific literature which are convenient but untrustworthy?

    The culture (certainly) and nature (possibly) of mathematical research is very different from that of the sciences. One difference that is particularly relevant here is that a “convenient” result—call it Theorem T—, even from a source, say Author A, that is known to be “untrustworthy”, can be the basis of trustworthy results. At a minimum, the conclusion of Theorem T can be called “A’s Conjecture” (politely), and trustworthy mathematical practice can produce correct theorems of the form “A’s Conjecture implies Claims U, V, and W”. If U or V or W are themselves interesting, then mathematicians (some or many, mediocre or of genius, depending on the details of “interesting”) may study them from other angles: if such study leads to proofs (not depending on A’s Conjecture) of U or V or W, then mathematics has been advanced; if it leads to a proof that U, or V, or W is false, then A’s Conjecture has been disproved (without necessarily finding the flaw or flaws in A’s purported proof of the conjecture).

    It was said (somewhat hyperbolically) of one very great early-to-mid 20th century mathematician, Solomon Lefschetz, that “he never stated a false theorem, and never gave a correct proof.”

  4. And in addition, this paper makes a mockery of the peer-review process. The ‘reviewers’ chosen by the authors do not have the subject matter expertise to evaluate this paper – the focus of this paper is not on atmospheric or aerosol science, but rather on epidemiology.

    1. Since when do authors pick their reviewers? Haven’t heard that before. On the contrary, reviewers are normally anonymous/undisclosed to the authors.

  5. The peer review process requires finding reviewers who are qualified, willing, and available. For urgent COVID-19 papers, that means, available NOW. Tough combination of requirements. Even tougher now that universities (where many reviewers have their day jobs) are responding to financial pressures by loading profs with more courses, burying them in forms having to do with compliance with this and that, and forcing them to redesign their courses for online delivery – all when the profs are working from home, with spotty Internet access and noisy kids. This appears to me to be the root cause of declining quality of peer review. Faculty and their unions should be making lots of noise about this in front of the university president’s office.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.