French hydroxychloroquine study has “major methodological shortcomings” and is “fully irresponsible,” says review, but is not being retracted

A March 2020 paper that set off months of angry debates about whether hydroxychloroquine is effective in treating COVID-19 has “gross methodological shortcomings” that “do not justify the far-reaching conclusions about the efficacy of hydroxychloroquine in Covid-19,” according to a review commissioned by the journal that published the original work.

The comments, by Frits Rosendaal, of Leiden University Medical Center in the Netherlands, came as part of a review commissioned by International Society of Antimicrobial Chemotherapy (ISAC), which publishes the journal along with Elsiever. ISAC had issued a statement about the paper in April, saying it “does not meet the [International Society of Antimicrobial Chemotherapy’s] expected standard.”

The study, Rosendaal writes,

suffers from major methodological shortcomings which make it nearly if not completely uninformative. Hence, the tone of the report, in presenting this as evidence of an effect of hydroxychloroquine and even recommending its use, is not only unfounded, but, given the desperate demand for a treatment for Covid-19, coupled with the potentially serious side-effects of hydroxychloroquine, fully irresponsible.

Rosendaal lists ten serious problems with the study, then concludes:

This is a non-informative manuscript with gross methodological shortcomings. The results do not justify the far-reaching conclusions about the efficacy of hydroxychloroquine in Covid-19, and in the view of this reviewer do not justify any conclusion at all.

In another commentary in the journal, Heiman Wertheim and colleagues at Radboudumc in the Netherlands conclude:

To date, robust clinical evidence of the efficacy of (hydroxy)chloroquine is lacking, let alone the combination with azithromycin. The paper by Gautret et al. raised a lot of attention and contributed to a demand for the drug without the appropriate evidence of its benefit. The study by Gautret et al. showed important methodological issues and does not provide a suggestion of effectiveness. A lack of COVID-19 study subjects and a strong motivation to find a treatment is not an issue, but good quality studies are needed. The International Society of Antimicrobial Chemotherapy provided an official statement on the paper in their journal that it did not meet their standard [54]. They stated that ‘it is important to help the scientific community by publishing new data fast, this cannot be at the cost of reducing scientific scrutiny and best practices’. The authors of this paper fully agree.

In an editorial, a group of ISAC’s senior officers explained the process, and why they chose not to retract the paper:

To minimize potential bias, as one of the authors is the Editor in Chief of the journal, the editorial handling / peer review was delegated to an Associate Editor. Furthermore, after publication ISAC arranged an additional independent peer review to ascertain whether concerns about the research content of the paper had merit.

After internal and external (post-publication) review, some of the concerns regarding the paper’s methodology were substantiated [6]. Consequently, the ISAC Executive Committee, in collaboration with Elsevier, discussed a variety of options in regard to how to proceed. Despite the flaws in methodology, we have elected not to withdraw the publication by Gautret et al [3]. We believe, in addition to the importance of sharing observational data at the height of a pandemic, a robust public scientific debate about the paper’s findings in an open and transparent fashion should be made available.

Prior to publication, the authors were invited to respond within the journal to this editorial and the reviews by Machiels et al. [2] and Rosendaal [6]. At time of publication, a response has been posted on the authors’ institutional website and is currently under review in IJAA.

The response by Raoult and colleagues is available here.

There have been other studies of hydroxychloroquine and COVID-19, a number of which have come under scrutiny. Most recently, a team at Henry Ford Hospital in Detroit claimed that their findings showed the drug was effective. However, as Matthew Herper at STAT noted:

The study that sparked the latest controversy was anything but randomized. Not only was it not randomized, outside experts noted, but patients who received hydroxychloroquine were also more likely to get steroids, which appear to help very sick patients with Covid-19. That is likely to have influenced the central finding of the Henry Ford study: that death rates were 50% lower among patients in hospitals treated with hydroxychloroquine.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].