Too many papers cite retracted research — even after it’s been retracted. It’s a problem. It can be especially a problem in clinical fields, where patient care is at stake. Recently, Richard Gray at La Trobe University in Australia and his colleagues examined the scope of the problem in the nursing field, noting how many systematic reviews included findings from retracted clinical trials. We spoke with Gray about their findings, published in the International Journal of Nursing Studies — and what they might mean for the safety of patients.
Retraction Watch: Retractions are a concern in any field, but as you note, when clinical practice is at stake, it can be particularly worrisome. Do you think your findings raise any potential concerns about patient safety?
Richard Gray: We identified 23 reviews that included a retracted trial. In a clinical discipline such as nursing, practitioners rely heavily on systematic reviews to inform their care and treatment decisions. If a review includes a study that is retracted, the integrity of the review must be challenged. There is no established mechanism to do this. As a consequence, clinicians may continue to base decisions on evidence from systematic reviews that is unsound. This is a concerning – largely unrecognised – threat to patient safety.
RW: You don’t list the reviews that cite retracted trials so you don’t “name and shame” them. But given that these papers do affect patient safety, do you see an argument for making that information public? How have readers reacted to your decision to not name the affected reviews?
RG: To date we have not had any feedback from readers about our decision not to include a list of included reviews. We did consider carefully if we should publish a list of the 23 systematic reviews we identified that included a retracted trial. And yes, we agree there is a strong case to publish the list, because ultimately these reviews impact on clinical nursing practice. In the spirit of transparent reporting, publishing would also enable readers to check the accuracy of our work. However, the aim of this study was to test a hunch; that retracted trials get included in systematic reviews (before and after they are retracted). Authors of the five reviews that cite already retracted trials had clearly made an error and we felt that the most appropriate course of action was for us was to write to journal editors informing them of our observation. We were less clear about how to deal with the 18 reviews we identified that included a trial where it appeared that the retraction happened after the review was published. Whilst authors may not have made an error, we felt that we still had a duty to write to these editors informing them of our observations. We did this after the publication of our study.
RW: Have you gotten any responses from the journals you’ve contacted regarding the reviews that cite retracted trials? If so, what have they been?
RG: On the 7th December 2017, we wrote to the editors of journals that published a review that included an already retracted trial; four of five responded. A follow up email was sent to the sole editor who did not respond. Three editors indicated that they would investigate the matter further. The fourth editor informed us that they intended to inform the authors of the review so they were aware. They went on to state that “of course, we can’t control retractions but the authors should definitely be aware.” This response did concern us as it seemed to indicate a failure to understand how to respond to the report of a significant error in a paper published in their journal. We thought we should check the status of the five manuscripts a month after writing to editors. Perhaps not surprisingly, no expressions of concern, erratums, or retraction notices have appeared against any of the five papers.
RW: You note an interesting question: Should a review that cites a retracted paper be retracted? Some journals have taken that step in the past. What is your opinion in this matter?
RG: Where review authors included an already retracted paper this is clearly an error and the quality and integrity of the review is compromised. It seems to us that retraction would, generally, be appropriate. Of course, this is a decision that individual journal editors have to make, in consultation with authors (and potentially author employers and study funders). Where retraction of a trial occurred after the review was published, authors have not made an error. That said, retraction of an included trial does still challenge the integrity of the review. Whilst it may seem harsh on review authors, there remains a compelling case that reviews should still be retracted. Ultimately, whether a researcher or practicing nurse, patient safety must be a primary concern and unsound reviews cannot continue to inform clinical practice, and consequently should be removed from the evidence base.
RW: Five reviews cited retracted clinical trials after they’d been retracted. This is a problem we’ve noted as well (and a major reason for our database). What do you think can be done to improve awareness of retractions among review authors (and the editors and reviewers who check their submissions)?
RG: We have given careful consideration to practices that could be adopted by authors, reviewers and editors to minimise the inclusion of retracted clinical trials in systematic reviews.
- A retraction filter (or whatever mechanism the database in question allows) must be applied to the end output of any search strategy.
- Journals/databases must make retractions more visible (step 1 above depends on it).
- Collaborations (e.g. Cochrane, Campbell, The JBI) need to incorporate into their handbooks directives around retraction. For example, a scan for retractions after data sourcing; a scan for retractions before data extraction; a scan for retractions before review submission.
- The reporting guidelines for systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PRISMA) needs to include an item stating that authors have checked if any included studies have been retracted.
- Journal editors should require authors, when submitting manuscripts, to confirm that they have checked that none of the included studies have been retracted. Authors should also include a statement in the paper stating they have done this.
- Proofreaders may also have an important role to play. For example, authors of one review included in their reference list a citation that clearly indicated the reference was for a retracted paper. Proofreaders could be trained to spot and report these anomalies.
We hope our work in this space helps create awareness of this important issue.
Additional point: There is – as far as we are aware – no mechanism for alerting journal editors that a paper in a review they have published has been retracted. We argue that retracting editors have a duty to the scientific community (and in clinical disciplines to patient safety) to review the citations of retracted papers, identify any (systematic) reviews, and then write to editors who published the review informing them of the retraction.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here. If you have comments or feedback, you can reach us at [email protected]
Richard Gray’s recommendations make sense. IF they are adopted, systematic reviews in the future will be less likely to contain retracted publications. But what about the instant case? Regardless of consequences to authors and publishers, should not the 23 trials that have been detected to contain retracted trials–whether or not retracted before or after publication of the systematic review–be made known in order to alert users of possible dangers? Preventing possible harm to patients should be the ascendent consideration in the decision, don’t you think? First do no harm!
This also highlights why proposals to eliminate retractions altogether, as advanced by some in the COPE leadership, are dangerous.
Only a retraction sends an unambiguous message that a study has a fatal flaw.
An amendment or correction allows misleading or overtly fraudulent papers to continue to disrupt the scientific literature esssentially unabated and eliminating retractions would provide a boon to the sloppy and unscrupulous few.
For clarity, COPE’s position on retractions is expressed in the COPE Core Practice on post-publication discussions and corrections. This says “journals must allow debate post-publication either on their site, through letters to the editor, or on an external moderated site, such as PubMed Commons or PubPeer. They must have mechanisms for correcting, revising or retracting articles after publication”. https://publicationethics.org/postpublication
Thank you for the clarification.
For the sake of further clarity my comment about some in COPE leadership wanting to do away with retractions is accurate.
Some from COPE leadership first proposed doing away with retractions in a preprint here:
https://www.biorxiv.org/content/biorxiv/early/2017/03/20/118356.1.full.pdf
This preprint was discussed here:
http://retractionwatch.com/2017/04/04/problematic-papers-dont-retract-correct-say-publishing-experts-amend/
An updated version was published here:
https://f1000research.com/articles/6-1960/v1
From the f1000 article:
“Competing interests
VB is the Director of the Australasian Open Access Strategy Group. She also works part-time for Queensland University of Technology (QUT), Brisbane as a Professor in the Office of Research Ethics & Integrity and in the Division of Technology, Information and Library Services. She was the Chair of COPE until May 2017 and was a COPE Trustee until November 2017. She is also an Editorial Board Member for Research Integrity and Peer Review. TB is Executive Editor of The BMJ. She chairs the scientific advisory board of EMBL-EBI Literature Services. JL is Director of Product Management at Crossref. She serves on the Dryad digital repository board. ECM is Senior Editor (Peer Review Strategy and Innovation) at BMC (part of Springer Nature). She is also a COPE Council Member, an Editorial Board Member for Research Integrity and Peer Review, a member of the Advisory Board for EnTIRE (an EU proposal for Mapping the research ethics and research integrity framework) and mentor for MiRoR (Methods in Research on Research).”
“Provenance
This work was conducted as part of an ongoing open discussion within an initial working group and a wider consultation. COPE convened the working group that wrote this paper. This is one of the ways that COPE helps foster high-quality and progressive discussions that, one day, may be reflected by changes in practice. This paper and the ideas presented within it exemplify COPE’s commitment to collegial discussion and debate.
Acknowledgments
We are grateful to COPE for facilitating this discussion. We are also grateful for comments and feedback on the preprint from Geoffrey Bilder, Peter Doshi, Andy Collings, Michaela Torkar and Liz Wager. We are also grateful to others who commented on the preprint versions1 and have attempted to incorporate the feedback into this paper.”
It’s clear that the proposal to do away with retractions is being actively discussed and considered at the highest levels of the COPE leadership [and elsewhere, like BMC, part of the Nature publishing group].
My opinion is that this is an extremely dangerous idea that should be rejected by every scientist who is concerned about scientific integrity and the reliability of the scientific literature.
In response to Richard Gray’s comment that “Collaborations (e.g. Cochrane, Campbell, The JBI) need to incorporate into their handbooks directives around retraction.”, I should like to point out that in Cochrane, we have been urging caution around retractions in the Cochrane Handbook since at least 2011, if not before. (I have not gone back to earlier editions.) The guidance offered in the 2011 (current) version is under revision for the Handbook update due for publication later this year.
Section 6.4.10 Identifying fraudulent studies, other retracted publications, errata and comments
Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from http://handbook-5-1.cochrane.org/
Let me restate and correct my original question, “Should not the titles of the 23 meta-analytic reviews containing a retracted trial, along with the titles of the retracted trials, be identified in order to alert readers of the possible danger in their use as a guide to clinical practice?” There is a way of using this information directly to correct the published record without reliance on the authors or the journal editors and publishers.
If I had the information to which Richard Gray is privy, I would have the basis for reanalyzing the meta-analytic reviews to determine the extent to which they are flawed. Are they entirely worthless or do they have partial value? Can they be weighted (zero = worthless) in redoing the the meta-analytic statistics and in conducting sensitivity analyses? This done, it would be possible to publish the results of the reanalysis in order to enable would-be users of the 23 meta-analytic reviews to judge for themselves their trustworthiness and possible marginal utilities.
My concern here is Richard Gray’s failure to make an outright commitment to disclosing the identities of the 23 meta-analytic reviews and those of the retracted studies they contain.
Had I been interviewing Richard Gray, I would have challenged concealment of available information as a violation of the precautionary principle, “First do no harm.” In my opinion, withholding information by which to take corrective action violates the norms of both deontological and consequentialist ethics.
Clearly not including retracted is important, but we perhaps should be careful about over-emphasising including retracted data can be.
For example, in the case of Scott Reuben, Martin Tramer’s Geneva group did a nice analysis that came to a lagely benign conclusion (see Anesthesiology. 2009 Dec;111(6):1279-89. doi: 10.1097/ALN.0b013e3181c14c3d. Susceptibility to fraud in systematic reviews: lessons from the Reuben case). The conclusion was that: “Carefully performed systematic reviews proved robust against the impact of Reuben reports. Quantitative systematic reviews were vulnerable if the fraudulent data were more than 30% of the total. Qualitative systematic reviews seemed at greater risk than quantitative.”
Of course, the issue is about quantity and proportion. Fujii wasa different matter. Again Tramer has a fun read about how the Anaesthesiolohy community dealt with this in the days before Retraction watch “Eur J Anaesthesiol. 2013 May;30(5):195-8. doi: 10.1097/EJA.0b013e328360a0db. The Fujii story: a chronicle of naive disbelief.”
Thank you, Carol, for your comment. Whilst section 6.4.10 of the handbook does identify the need for caution around, and certain methods by which to identify, retracted studies, it does not address what the inclusion of a retracted paper “means” for the integrity-for the epistemic value-of a systematic review. This is the conversation in which we want researchers to engage. Given a systematic review is essentially an observational study whose data are the studies it synthesises, and a removed datum post-publication has the potential of altering the magnitude of effect, it seems prudent for the collaborations to elaborate on the implications of retraction-inclusion and perhaps consider a repertoire of directives in response to its occurrence.
Thank you. I shall ensure that the Editors of The Cochrane Handbook are aware of this discussion as we prepare the major revision for publication later this year.
Thank you, Carol. I look forward very much to the new edition.