Research integrity during the COVID-19 pandemic: A book excerpt

We are pleased to present an excerpt from Thinking About Science: Good Science, Bad Science, and How to Make It Better by Ferric C. Fang and Arturo Casadevall, published by ASM Press/Wiley, October 2023.

Amidst the COVID-19 calamity, one can argue that science is one of the few aspects of the human response that has worked relatively well. However, despite the many advances in preventing and treating COVID-19, there have also been missteps as the world has scrambled to respond to a deadly new pathogen. It has been humbling for the U.S. to lead large high-income countries in per capita deaths from COVID-19 even with its wealth and scientific expertise. We are all too aware of the needless illnesses and deaths that have resulted from misguided political leadership, inadequate preparation, delayed responses, fragile supply chains, health disparities, and vaccine hesitancy. But we will not dwell on these issues here. Rather, we would like to review the COVID-19 pandemic through the prism of the 3R’s of research integrity: rigor, reproducibility, and responsibility. These form the fundamental pillars of the foundation of science. It is appropriate that we devote more attention to the foibles than to the successes so that we can learn from the mistakes and missed opportunities. What could have been done better? What needs to improve?

Has COVID research been rigorous and reproducible?  Certainly, much of the work that brought us vaccines and new therapies has been. But unfortunately, other instances have fallen far short of rigor. Perhaps the most prominent example early in the pandemic related to the use of hydroxychloroquine for the treatment of COVID-19. Chloroquine and hydroxychloroquine, drugs used for the treatment of malaria and rheumatic diseases, were suggested as possible therapies for SARS in 2003, but were not formally studied in a clinical setting. Shortly after the COVID pandemic began to take off in China, rumors of a possible benefit of these agents began to appear in social media. On February 19, 2020, a letter was published in an obscure Asian journal, which mentioned a press briefing reporting that chloroquine improved clinical outcomes compared to control treatment in more than 100 patients with COVID-19 in China. No data were provided however, and the paper initially attracted little notice. However, on March 4, an article from a prominent research group in France led by Dr. Didier Raoult appeared in the International Journal of Antimicrobial Agents, reviewing the in vitro antiviral activity of chloroquine against coronaviruses and referring to the reportedly favorable experience in China.

On March 13, two cryptocurrency investors, Greg Rigano and James Todaro, posted a non-peer-reviewed paper online, touting chloroquine as “an effective treatment for COVID-19.” The authors claimed some prestigious institutional affiliations, which were subsequently disavowed, but nevertheless their paper was tweeted by Elon Musk to over 40 million followers, and one of the authors was interviewed by Tucker Carlson of Fox News, who publicized the claims to millions of viewers. On March 16, a preprint from Dr. Raoult and his group reported the use of hydroxychloroquine with or without azithromycin in 26 patients, who were compared with 16 controls receiving standard care. Viral clearance as measured by PCR was reported to be more rapid in recipients of hydroxychloroquine and much more rapid in those who also received azithromycin. The paper was accepted for publication soon afterward.

Red flags about this paper were raised almost immediately, but not before hydroxychloroquine was being touted by President Donald Trump as a “game changer” that would be fast-tracked by the FDA for approval. On March 28, the FDA issued Emergency Use Authorization for hydroxychloroquine and chloroquine in patients with severe COVID-19. Among the concerns regarding the French study were the small sample size, the lack of randomization (which resulted in poorly matched study and control groups), the use of different diagnostic assays, and the failure to account for six patients who had been initially enrolled in the study. It was also noted that the Editor-in-Chief of the journal publishing the article was one of the authors, and the review process took less than 24 hours. The same research group subsequently published a paper reporting that hydroxychloroquine and azithromycin were 92% effective in more than 1,000 patients with early COVID-19. However, the follow-up study did not include a control group. Dr. Raoult was quoted as saying that randomized controlled trials are unnecessary and unethical in deadly infectious diseases and appeal only to statisticians “who have never seen a patient.” Meanwhile, he initiated a lawsuit against Dr. Elisabeth Bik, one of the scientists who had criticized his original paper on hydroxychloroquine.

We now know from numerous subsequent clinical studies involving thousands of patients that hydroxychloroquine, with or without azithromycin, is not beneficial for patients at any stage of COVID-19, nor for the prevention of infection, and may even be associated with a higher risk of death. With very few exceptions, the results are highly consistent. The FDA Emergency Use Authorization was withdrawn on June 15, 2020. Yet hydroxychloroquine has been given to countless patients on the basis of non-rigorous science and continues to be given in some settings to this day.

As if to prove the adage that history repeats itself—the first time as tragedy, the second as farce, no sooner had enthusiasm for hydroxychloroquine begun to wane, when a new unproven treatment began to gain rapidly in popularity—ivermectin. Ivermectin is a macrocyclic lactone used to treat parasitic infections in humans and other animals. Early in the pandemic, Australian scientists reported that ivermectin could inhibit the replication of SARS-CoV-2 in vitro. It was quickly pointed out that the concentration of ivermectin required to inhibit viral replication greatly exceeded concentrations achievable from normal human dosing of the drug, but unfortunately this warning was unheeded.

Soon there were reports that ivermectin could prevent SARS-CoV-2 infection or reduce disease progression in patients with mild-to-moderate COVID-19. A systematic review found that most clinical studies of ivermectin failed to meet predefined eligibility criteria and suffered from imprecision and a high risk of bias. The authors of the review concluded that it is uncertain whether ivermectin is beneficial in COVID-19. But this message was drowned out by ringing endorsements from groups like “America’s Frontline Doctors,” a group of physicians closely aligned with right-wing political organizations, which had originally advocated the use of hydroxychloroquine to treat COVID. A related group calling themselves the Front Line COVID-19 Critical Care Alliance, or FLCCC, led by a critical care physician in Wisconsin named Pierre Kory, lent their voices to these efforts. Dr. Kory was quoted as saying, “My dream is that every household has ivermectin in their cupboard. And you take it upon development of the first symptom of anything approximating a viral symptom. . . Even if it’s not COVID, it’s safe to take it and it’s probably effective against that virus.”

During that latter part of the summer of 2020, an explosive rise in ivermectin prescriptions was observed in the US. High doses of ivermectin can produce a variety of adverse effects including gastrointestinal symptoms, seizures, respiratory failure, and coma. Poison centers found themselves swamped with calls about ivermectin overdoses. The FDA even issued a warning not to take ivermectin for COVID, noting that some people were even taking ivermectin preparations intended for the deworming of horses. Off-label ivermectin use continued to be popular on social media even though careful reviewers found serious flaws in the purported evidence for its benefits in COVID. An independent analysis failed to find a single clinical trial demonstrating a benefit from ivermectin in COVID that did not contain “either obvious signs of fabrication or errors so critical that they invalidated the study,” including the same patient data being used to represent multiple subjects, non-random patient selection, numbers unlikely to occur naturally, incorrectly calculated percentages, and an inability of local health organizations to corroborate that the studies took place.”

Were these aberrations? Extreme cases perhaps, but systematic analyses suggests that deficiencies in the rigor of COVID-related research have been commonplace. A study of 686 COVID-19 clinical research articles found shorter time to publication and lower methodological quality than for other articles in the same journals. This suggests a lowering of the usual standards for publication in the time of plague.

Another issue that cannot be ignored is the important role of social media in disseminating and amplifying misinformation. A commentary has discussed an urgent need for improved understanding and stewardship of global collective behavior, cautioning that it has become easy to connect and share information through social media, but “in contexts where decisions depend upon accurate information, such processes can undermine collective intelligence and promote dangerous behavior.” A lack of trust in government institutions is undermining public health efforts. A country-level analysis demonstrates an inverse correlation between confidence in institutions and COVID burden. As the epidemiologist Jay Kaufman observed, “Science alone can’t heal a sick society. . . Science is a social process. . . To restore faith in science, there must be faith in social institutions.” Our social divisions are literally killing us.

According to Retraction Watch, more than 350 articles relating to COVID-19 have been retracted at the time of this writing. This is a modest number in view of the nearly 400,000 articles published on COVID, but it is nevertheless instructive to look at a few of the most high-profile retractions. Two of the retracted articles were published in the most prestigious and selective clinical journals in the world, the Lancet and the New England Journal of Medicine.

The Lancet paper, published in May 2020, purported to describe a multinational registry analysis of 96,032 patients hospitalized with COVID and found that hydroxychloroquine and chloroquine were associated with decreased in-hospital survival. The New England Journal of Medicine paper, published the following month, purported to analyze 8,910 patients hospitalized with COVID-19 and found no association between the use of angiotensin converting enzyme inhibitors or angiotensin receptor blockers and in-hospital mortality. Note that both of these findings have been subsequently confirmed by other legitimate studies, so what is in dispute is not the bottom-line findings of these articles. The problem is that the data were made up out of thin air. The retracted articles claimed to draw from an enormous registry including patients from 671 hospitals on six continents. One would expect such a monumental effort to involve hundreds if not thousands of contributors. However, suspicions quickly arose when no one could identify even a single hospital that had contributed to this registry, and major discrepancies were noted between the number of reported cases for some regions and data from independent sources.

Although both papers were retracted within weeks of publication, significant damage was done. Clinical trials had been suspended and international guidelines had been revised on the basis of the published findings. Editors and reviewers were apparently seduced by the reputation of the first author (a respected Harvard Professor) and the power of big data. The pressures of the pandemic may have allowed the papers to slip through the peer review process with less than the usual scrutiny. Dr. Sapan Desai, the founder of Surgisphere, the company that claimed to have assembled the registry, turned out to be a surgeon with multiple prior malpractice claims and little experience in data analytics. An investigation of his company identified just a handful of employees, several of whom had no background in science or data analysis, and the company was occupying rented office space. Dr. Amit Patel, a co-author on the studies, turned out to be Dr. Desai’s brother-in-law. The first author Dr. Mandeep Mehra admitted to not having seen any primary data. Although Dr. Desai has not admitted any wrongdoing, the Surgisphere papers are now believed to have been a hoax based on fabricated data. The editor-in-chief of the Lancet concluded that the study was a “monumental fraud.”  Nevertheless, the papers have continued to be cited despite their retraction; each has been cited more than 700 times, and many of the citing articles appear to be unaware that the papers have been retracted.

Further evidence of peer review failure can be seen in two retracted articles by a psychologist named Harald Walach. Walach published articles claiming that COVID vaccines were responsible for two deaths for every three infections prevented and that face mask use results in hypercapnia in children. Although the papers were strongly criticized for their nonsensical findings and quickly retracted by the journals Vaccines and JAMA Pediatrics, they were widely circulated and continue to be cited by anti-vaccine and anti-face mask advocates.

Serious questions regarding scientific responsibility have arisen during the pandemic. Although SARS-CoV-2 was initially assumed to have arisen as a zoonotic pathogen making a species jump from bats to humans, possibly via an intermediate host, subsequent attention was focused on the possibility of transmission from a lab leak involving scientists at the Wuhan Institute of Virology, where coronavirus research is performed. Speculations were made about whether so-called “Gain-of-Function” research to identify viral adaptations that facilitate human infection might have inadvertently led to the emergence of SARS-CoV-2. At this time, the weight of scientific evidence favors a natural origin. Nevertheless, the lack of transparency on the part of the Chinese government and the politicization of the issue have raised uncomfortable questions about whether scientists are adequately considering and mitigating the public health risks of research with dangerous pathogens. As Dr. Tedros Ghebreyesus, the Director-General of the WHO has noted, “Lab accidents happen,” and it is incumbent upon scientists to be cognizant of these hazards and to take every possible precaution to minimize the risk to the public. In his classic description of the Black Death in London, A Journal of the Plague Year, Daniel Defoe wrote “It mattered not from whence it came.” But the origin of the pandemic does matter. Whether by zoonotic spillover or lab accident, we need to understand how the pandemic began in order to take appropriate measures to prevent this from happening again.

Although we have focused on some examples in which science fell short of its ideals during the COVID pandemic, overall, the scientific community acquitted itself well. We have already mentioned the remarkable success of vaccine development efforts. Scientists threw themselves wholeheartedly into COVID research, putting their other interests on hold. Collaborative and team science allowed the rapid execution of multinational trials, and data sharing, open access and preprint servers facilitated the timely dissemination of information. Critical knowledge regarding pre-symptomatic spread and superspreading, non-pharmaceutical interventions, diagnostic testing, distinction between protective and detrimental immune responses, ventilatory management, thromboembolic complications, and viral variants was quickly and widely shared, allowing substantial improvements in clinical outcomes as the pandemic proceeded.

Nevertheless, both the strengths and weaknesses of the contemporary scientific enterprise have been on ample display. Scientists produced a massive number of publications, but wading through this literature to find the truth has not been easy. As Robert Peter Gale wryly observed, the pandemic would be quickly ended if SARS-CoV-2 could be conquered by publications, guidelines, and interminable meetings alone. Due to the urgency of the pandemic, scientific standards were sometimes compromised. Limitations of randomized clinical trials as an exclusive source of knowledge to guide clinical decision-making were exposed. Important questions were raised about the potential dangers of scientific research on pathogenic microbes, while the origins of the pandemic have remained uncertain.

Vigorous debate is often regarded as a sign of a healthy scientific enterprise. However, open bickering among scientists about public health policies instead contributed to increasing societal polarization and undermining of public trust in science during the pandemic. Peter Sandman, an expert on risk communication, identified many mistakes made by officials in their messaging to the public, including a failure to communicate uncertainty and acknowledge error. Resistance to lockdowns and mandates were frequently met by backlash and hostility to public health officials. Unfortunately, the loss of credibility and public resistance to expert guidance have proven persistent, even as pandemic waves have receded.

Social media has posed a formidable challenge by creating alternative sources of information and disinformation that compete with science in influencing policy-making and public opinion. This represents a serious threat that society has scarcely begun to address. With science as just one among many potential sources of information, the imperative of making science as rigorous, as reproducible, and as responsible as possible becomes clear. Otherwise, why should anyone believe scientists rather than other sources claiming to represent the truth?  For science to retain its influence in society, efforts to make it more rigorous, reproducible, and responsible must be a ceaseless undertaking. Only scientists can carry the torch and continue to perform the high-quality research that will lead to better prevention and treatment, not only of COVID, but of future threats as well.

As Albert Camus observed in The Plague, “What’s natural is the microbe. All the rest—health, integrity, purity—is a product of the human will, of a vigilance that must never falter.” We can and must learn from our experience because it is a certainty that COVID will not be the last pandemic.

Ferric C. Fang is a Professor of Laboratory Medicine and Pathology, Microbiology, Medicine, and Global Health at the University of Washington School of Medicine, Director of the Clinical Microbiology Laboratory at Harborview Medical Center, and a former Editor-in-Chief of Infection and Immunity. He is a member of the board of directors of The Center For Scientific Integrity, Retraction Watch’s parent nonprofit organization.

Arturo Casadevall is a Bloomberg Distinguished Professor and Chair of Molecular Microbiology and Immunology at the Johns Hopkins Bloomberg School of Public Health, a Professor of Medicine at the Johns Hopkins School of Medicine, and the Editor-in-Chief of mBio.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in The Retraction Watch Database, you can let us know here. For comments or feedback, email us at [email protected].