Cochrane has implemented a new system for checking whether any of its thousands of published reviews include retracted studies in their analyses, the organization announced today. The effort already has turned up dozens of reviews that do and will now get closer scrutiny to ensure their results and recommendations hold up.
Cochrane publishes systematic reviews on health-related topics that evaluate the strength of evidence on particular treatments and interventions. Professional organizations and policymakers use the more than 9,500 reviews when developing recommendations. Recently a study of anesthesia clinical trials found a high rate of the studies with faked or flawed data, and another revealed that retracted studies included in systematic reviews — 17% of which were Cochrane reviews — had a large impact on clinical guidelines derived from them.
Last year, Cochrane rolled out a feature in its database of reports of clinical trials, called CENTRAL, to flag retracted studies. The publisher pulls data on retracted papers from the Retraction Watch Database, via CrossRef. Now, they have extended the process to routinely identify systematic reviews that rely on retracted papers.
“What we are looking at is studies that have been retracted that contribute results or findings to the conclusions of the Cochrane review,” Ella Flemyng, Cochrane’s head of editorial policy and research integrity, told Retraction Watch. “So if a study was already excluded, or it was just a general reference in the introduction or the discussion, we didn’t want to prioritize those.”
The publisher has developed a workflow for assessing the impact of the retracted work on the systematic review, also announced today alongside guidance for implementing it.
“Cochrane has very rigorous methodological standards for developing systematic reviews in terms of identifying studies that have been retracted while you’re developing the review itself,” Flemyng said. “The gap that we had is when retractions happen after the publication of the systematic review. This new framework allows us to inform readers as quickly as we become aware of this impact on the evidence.”
Two published reviews served as test subjects while Cochrane developed that framework. One review, on the effects of antioxidants on reduced fertility, was flagged in December 2023 by the Feet of Clay Detector for containing four retracted papers. Cochrane placed an editorial note on the review a year later. In total, seven studies included in the review have been retracted since it was published, with another two studies marked with expressions of concern. Using Cochrane’s new framework, the authors evaluated how each study contributed to the overall analysis and the effect it had on the assessment.
“In the next update of this Cochrane review, these nine studies will be excluded from the analyses,” the March 5 editorial note states. “The studies with retractions will be moved to the excluded studies table and the studies with expressions of concern will be moved to studies awaiting classification.”
Similarly, a 2016 review on antiestrogens for improving fertility in women with polycystic ovary syndrome — now called polyendocrine metabolic ovarian syndrome — included four studies that have since been retracted. Following the framework, the authors determined removing them would not change the overall findings of the review. A May 22 editorial note includes details about each of the retracted studies.
“It’s important that readers will know when reviews have been checked — and the protocol does that,” said Jodi Schneider of the University of Wisconsin–Madison, who studies the persistence of retracted papers in the published literature. “Publishing an editorial notice indicating that the review has been checked, even if authors find minimal impact, is important.”
In checking all studies included in all systematic reviews, Cochrane found “just under 1%” of its 9,500 published systematic reviews include retracted papers in their analyses, Flemyng said. Now that the publisher has developed and tested the new framework, the staff will begin notifying the authors of those reviews about the retracted studies and encouraging them to follow the framework to determine the impact on the review’s recommendations.
Cochrane will also start putting editorial notes on reviews as soon as they are flagged for including retracted papers. Once authors have completed a review as outlined in the new framework, Cochrane will update the editorial note.
In the future, Cochrane hopes to automate more steps in the process, such as adding an editorial note on affected reviews and notify authors more quickly, Flemyng said.
Schneider, who called the new framework “excellent,” suggested an area for future development could be to assess studies that have been flagged as problematic by INSPECT-SR or some other criteria but have not been retracted.
“Cochrane reviews inform guidelines, they inform clinical decisions, they inform health decisions. So we need to make sure that the underlying evidence remains reliable,” Flemyng said. “If studies are being retracted, we need to make sure the systematic reviews that are based on that evidence are also updated so that we can continue to have relevant, up-to-date evidence informing health and care-related decisions and policies.”
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].