When a paper is retracted, how many other papers in the same field — which either cite the finding or cite other papers that do — are affected?
That’s the question examined by a study published in BioMed Central’s new journal, Research Integrity and Peer Review. Using the case of a paper retracted from Nature in 2014, the authors found that subsequent research that cites the retracted paper often repeats the problematic finding, thereby spreading it throughout the field. However, papers that indirectly cited the retracted result — by citing the papers that cited the Nature paper, but not the Nature paper itself — typically don’t repeat the retracted result, which limits its spread.
Here’s how the authors describe their findings in the paper:
We show that directly citing articles is an important source of propagation of retracted research results. In contrast, in our case study indirect citations do not contribute to the propagation of the retracted result.
One limitation to the paper, which the authors readily acknowledge: They looked at citation chains that resulted from only one paper, “The NAD-dependent deacetylase SIRT2 is required for programmed necrosis,” that suggested that the enzyme sirtuin-2 — implicated in aging — is associated with cellular necrosis. As we reported in 2014, the authors retracted it after multiple groups could not reproduce some of the data.
The retracted paper — by Nisha Narayan et al — was the start of a citation network studied by Paul E van der Vet and Harm Nijveen, the study’s authors from the Netherlands Bioinformatics Centre and Wageningen University in the Netherlands, respectively. In a joint statement, they told us more about their method:
…we collected papers in two rounds that [we] will call 2014 and 2015, respectively. In each round we collected all papers that directly cite the Narayan paper, papers that directly cite those papers, and so on, until we arrived at papers that are not themselves cited. We did so exhaustively. We did not count rounds, we just followed each branch as it unfolded while we were searching.
They went on:
In this way, in 2014 we collected 187 papers of which 37 directly cite the Narayan paper; in 2015, we collected 1626 papers of which 57 cite Narayan directly. We read all papers that directly cite the Narayan paper in both rounds.
Another limitation to the study, the authors told us, is that a retraction of a Nature paper may also not be representative of retractions overall:
A paper in Nature will attract more attention, which is likely to result in more groups trying to reproduce its results and finding them irreproducible, leading to a relatively quick retraction (in the same prestigious journal). A less visible paper might take longer to be found out, which gives it more time to collect citations, and its subsequent retraction will also be less visible.
Furthermore, it’s possible that only one aspect of the Narayan paper is incorrect, the authors told us:
The reason for the retraction of the Narayan paper is that one of the main findings of the paper could not be reproduced by a number of other groups, who subsequently as a collective published that result in Nature. Narayan and co-authors felt obliged to retract their paper “in its entirety.” Two papers by yet another group published this year seem to rehabilitate Narayan but there still is the impressive negative result that prompted the retraction.
When asked why they picked the Narayan paper for their analysis out of the countless retractions available, the authors said they had “no thorough selection procedure,” adding:
It has a number of advantages: high visibility, also because of the editorial in the same issue of Nature, and the popularity of the subject. We reasoned that if you are looking for propagated content you better start with a paper that will be widely read. We also did not want to choose a paper that is retracted because of misconduct. But then still very many papers qualify. Well, we had to pick one.
Adrian Letchford, a data scientist at the University of Warwick in Coventry, UK, said:
I think this study clearly demonstrates that retracted results can stay within the body of literature through the citation network.
He added:
I would like to know if this is a problem with every retracted paper, or just a fraction of them…This would allow us to see how common this is, and find out if it is only a problem for prestigious journals, or for every journal.
Although Narayan et al retracted their paper in its “entirety,” it is possible that readers will continue to cite the “correct bits,” noted Letchford:
The authors even say in their paper that some studies cite Narayan’s methods rather than results.
The study, “Propagation of errors in citation networks: A study involving the entire citation network of a widely cited paper published in, and later retracted from, the journal Nature,” also addresses a major issue in the literature: Problematic papers continue to be cited long after they’ve been retracted, with little recognition of the retraction. Here’s what van der Vet and Nijveen note in the paper:
With two exceptions, both from the 2015 network, none of the directly citing papers shows any awareness of the retraction. Yet most papers of the 2015 network have been published well after the retraction was published.
Other research has produced similar findings: When John Budd analyzed 235 retracted papers between 1966-1999, he discovered they had accumulated more than 2,000 citations collectively, with fewer than 8% of papers recognizing the retraction. In another analysis of 1,112 papers retracted between 1997-2009, Budd found them to be cited to a similar extent, but the retractions are acknowledged in only around 4% of the cases. Regarding the present study, Budd told us:
If this kind of problem can occur in the most prestigious journals, then all of the journal literature is vulnerable to error and misconduct.
The study marks the launch of the journal, which is edited by Elizabeth Wager, a member of the board of directors of our parent non-profit organization.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
In my limited experience of following up a paper with a published retraction (N=1), I discovered that the part of the paper untouched by the retraction notice (quite detailed, in this case) was also wrong. Retraction notices are often opaque, and only very occasionally provide the sort of detail (figure by figure) that we are due. So I suspect that most retracted papers are likely wrong through and through.
I would like to see a study into how the conclusions from a ‘high-profile’ paper that gets retracted pervade in the media. We always see news outlets online and in print playing up conclusions from papers (we cured cancer, no more aging, you can eat anything you want and not get fat…), some of which ought to end up being false and retracted. Would like to see how the general public’s opinion changes based on these original articles that then get retracted. I doubt the media follows up on the retraction and ‘alerts’ the public as often as they over-step and over-conclude something ‘salient’ from a paper in the first place.