French hydroxychloroquine study has “major methodological shortcomings” and is “fully irresponsible,” says review, but is not being retracted

Frits Rosendaal

A March 2020 paper that set off months of angry debates about whether hydroxychloroquine is effective in treating COVID-19 has “gross methodological shortcomings” that “do not justify the far-reaching conclusions about the efficacy of hydroxychloroquine in Covid-19,” according to a review commissioned by the journal that published the original work.

The comments, by Frits Rosendaal, of Leiden University Medical Center in the Netherlands, came as part of a review commissioned by International Society of Antimicrobial Chemotherapy (ISAC), which publishes the journal along with Elsiever. ISAC had issued a statement about the paper in April, saying it “does not meet the [International Society of Antimicrobial Chemotherapy’s] expected standard.”

The study, Rosendaal writes,

suffers from major methodological shortcomings which make it nearly if not completely uninformative. Hence, the tone of the report, in presenting this as evidence of an effect of hydroxychloroquine and even recommending its use, is not only unfounded, but, given the desperate demand for a treatment for Covid-19, coupled with the potentially serious side-effects of hydroxychloroquine, fully irresponsible.

Rosendaal lists ten serious problems with the study, then concludes:

This is a non-informative manuscript with gross methodological shortcomings. The results do not justify the far-reaching conclusions about the efficacy of hydroxychloroquine in Covid-19, and in the view of this reviewer do not justify any conclusion at all.

In another commentary in the journal, Heiman Wertheim and colleagues at Radboudumc in the Netherlands conclude:

To date, robust clinical evidence of the efficacy of (hydroxy)chloroquine is lacking, let alone the combination with azithromycin. The paper by Gautret et al. raised a lot of attention and contributed to a demand for the drug without the appropriate evidence of its benefit. The study by Gautret et al. showed important methodological issues and does not provide a suggestion of effectiveness. A lack of COVID-19 study subjects and a strong motivation to find a treatment is not an issue, but good quality studies are needed. The International Society of Antimicrobial Chemotherapy provided an official statement on the paper in their journal that it did not meet their standard [54]. They stated that ‘it is important to help the scientific community by publishing new data fast, this cannot be at the cost of reducing scientific scrutiny and best practices’. The authors of this paper fully agree.

In an editorial, a group of ISAC’s senior officers explained the process, and why they chose not to retract the paper:

To minimize potential bias, as one of the authors is the Editor in Chief of the journal, the editorial handling / peer review was delegated to an Associate Editor. Furthermore, after publication ISAC arranged an additional independent peer review to ascertain whether concerns about the research content of the paper had merit.

After internal and external (post-publication) review, some of the concerns regarding the paper’s methodology were substantiated [6]. Consequently, the ISAC Executive Committee, in collaboration with Elsevier, discussed a variety of options in regard to how to proceed. Despite the flaws in methodology, we have elected not to withdraw the publication by Gautret et al [3]. We believe, in addition to the importance of sharing observational data at the height of a pandemic, a robust public scientific debate about the paper’s findings in an open and transparent fashion should be made available.

Prior to publication, the authors were invited to respond within the journal to this editorial and the reviews by Machiels et al. [2] and Rosendaal [6]. At time of publication, a response has been posted on the authors’ institutional website and is currently under review in IJAA.

The response by Raoult and colleagues is available here.

There have been other studies of hydroxychloroquine and COVID-19, a number of which have come under scrutiny. Most recently, a team at Henry Ford Hospital in Detroit claimed that their findings showed the drug was effective. However, as Matthew Herper at STAT noted:

The study that sparked the latest controversy was anything but randomized. Not only was it not randomized, outside experts noted, but patients who received hydroxychloroquine were also more likely to get steroids, which appear to help very sick patients with Covid-19. That is likely to have influenced the central finding of the Henry Ford study: that death rates were 50% lower among patients in hospitals treated with hydroxychloroquine.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].

10 thoughts on “French hydroxychloroquine study has “major methodological shortcomings” and is “fully irresponsible,” says review, but is not being retracted”

  1. Major methodological shortcomings, gross methodological shortcomings, important methodological issues, fully irresponsible, did not meet the journal’s standards, co-author Editor in Chief. Despite this, not retracted, not even an expression of concern. Incredible!

  2. I would note that the reaction to the lack of retraction may be greater than a reaction to a potential retraction. In both cases, the study is disparaged and discarded. But in the latter case, an uproar insues as to why the retraction wasn’t done. Which course is most headline eye-catching?

  3. Having just read the response by Raoult and colleagues, it seems to me that the groups’ findings on the efficacy of chloroquine have been largely replicated. Wouldn’t you opt to receive chloro if infected, given the body of evidence in favour of the drug?

    1. Latest publication in Nature says that the mechanism of action that was proposed in Vero kidney cells cannot be replicated in a lung epithelial line, so no.

  4. 26 patients is nothing more than a large case series. No serious statistical analysis needed beyond simple percentages.

    Yet the reviewer wants stuff like “sensitivity analysis” done.
    Now what EXACTLY would constitute “sensitivity analysis” in this case.

    I am a statistician. I am curious.

  5. I’m just waiting to see if the same care and attention to the fast developing vaccines will be deployed!

    Using young and healthy subjects and giving them paracetamol to reduce symptoms seems to be the right approach, especially when we supposedly want to protect the old folks!

    And since most of the consensus is that antibodies don’t last long… I wonder if they will do another AB count after at least 6 months?!

    But all is well in Vaccine Fairy Land…

  6. The decision not to retract is deplorable. Days after this paper came out, many countries generalised treatment with cq/hcq. I posted a youtube video in April ecplaining all the flaws of the paper for the population in one of those (with English caption) here:

  7. I’m also not sure if a retraction would be the best option. If the data is real, there are no errors in the analyses and no misconduct, then the “only” problem would be interpretation of the findings and conclusions of the study. If the Editors really (now) think that the conclusions are not supported by the findings, which I really think is the case here (36 cases is actually a small case series, completely unable to inform about treatment efficacy), maybe they should issue first an expression of concern and offer the authors the possibility to correct the paper. But I guess it will be another case of resistance to post-publication review so typical of some journals.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.