Retraction Watch readers are no doubt familiar with one of the most consequential retractions of this century, namely that of the 1998 paper in The Lancet by Andrew Wakefield and others claiming a link between vaccines and autism. What they may also know is that the paper remains one of the most highly cited retracted articles of all time, as demonstrated by our leaderboard of such papers.
With that in mind, six librarians from institutions in Wisconsin had a question: ” What are the characteristics of citations of the retracted 1998 article by Wakefield et al that purported to show an association between the measles-mumps-rubella vaccine and autism?” A paper describing their findings was published on Friday in JAMA Network Open.
We asked corresponding author Elizabeth Suelzer, of the Medical College of Wisconsin, to answer several questions about the paper.
Retraction Watch (RW): Why did you decide to focus on the Wakefield paper, and what were your findings?
Elizabeth Suelzer (ES): With the recent measles outbreaks around the United States and having observed a very vocal anti-vax movement on social media, my group was interested in learning more about the scholarly perceptions of the infamous retracted Wakefield paper. We were interested in examining who was citing the paper, the way they cited it (negatively or affirmatively), and whether researchers were documenting the retracted status of the article. Very soon after the article was initially published, researchers began finding significant flaws in the study design and noted that the results were not reproducible. After the retractions of the study, more of the articles that cited the Wakefield study focused on the role the study played in public perception and attitudes towards vaccines. All-in-all, a majority of the references to the Wakefield study were negational. It wasn’t very surprising that few authors cited the 2004 partial retraction because partial retractions are unusual. We were surprised to see that a significant number of authors weren’t referencing the retracted status of the paper after 2010. Even authors who used terms such as “flawed” or “false” to describe the Wakefield paper didn’t always note the retracted status of the paper. My team felt that documenting the retraction carries a great amount of weight in demonstrating that the findings were fraudulent, and by missing out on this important piece of information, people may be under the perception that the work could be valid. The retraction of the Wakefield study is very well known and if authors are failing to note the retracted status of this article, we are concerned that lesser known retracted articles are being cited without documenting their retraction.
RW: What role do continued citations of this paper play in public perceptions of vaccine safety? Are they similar to the role that a 1980 paper in NEJM — and that earned an editor’s note decades later — that downplayed the risk of opioid addiction has played over the years?
ES: My group read the letter by Leung et al with great interest, and we use it as an example when we teach evidence-based medicine. Our study was inspired by it.
We feel that the majority of researchers understand the importance of vaccines and can easily articulate why the Wakefield study was so flawed. But for those unfamiliar with the research such as students, those from other disciplines, and the public, the number of citations this retracted study receives can be misleading. There seems to be a disconnect between what occurs within the scientific community and how it is communicated and shared with the general public via social media. This is also evident in public perceptions of the threat of global warming and gun violence. Scientists and researchers need to do a better job of making their research findings easier to understand, emphasizing its relevance to the general public, and making it meme-worthy for social media.
While most of the references to the Wakefield article are negative, each new citation is noted in databases like Google Scholar, Web of Science and Scopus. As citation counts continue to play a role in determining the significance or importance of an article (for better or worse), even negative citations will ensure that an article gets a higher rank in databases when the results are sorted by citation count. We accept the irony of conducting a study on Wakefield’s paper and adding yet another count to its cited-by number.
RW: You write that “The International Committee of Medical Journal Editors (ICMJE) notes that not all journals check the accuracy of references in submitted publications.” Should that change?
ES: We acknowledge that it would be a complicated endeavor for journals to check the accuracy of references for submitted articles, but journals need to hold authors accountable for ensuring that their references are accurate.
RW: You note that “not all citation styles provide guidelines for citing retractions.” What would you recommend for such guidelines?
ES: As far as we know, the AMA style is the only citation style that explicitly states how to cite a retracted article in their manual. The APA gives some guidance on their website, but not in their manual. We feel that all citation styles should provide guidance on how to cite this type of reference. At the bare-minimum, guidelines should require the word “Retraction:” to appear before the article title of a retracted article.
RW: Although, for reasons you describe, it is often the case that authors do not know about retractions, in the case of the Wakefield paper, the retraction is reasonably well-known. What do you think leads to continued citations that do not note the retraction?
ES: We identified a number of barriers in citing retracted articles in our article, but we feel that the biggest barrier happens after downloading a retracted reference from the Internet into a citation manager and citing the reference in Word. For example, PubMed displays a banner on retracted articles making its retracted status very apparent. However, the “retracted” notice is buried when the citation information is imported into Zotero or EndNote, and it doesn’t show up in the automatically generated bibliography entry. Many authors are overly reliant on citation managers to create their reference lists and may not realize that information may be missing.
RW: You mention our partnership with Zotero, which “now includes an enhancement that will identify retracted articles that have been downloaded into a Zotero library,” but note that “not all citation management software includes this.” Would you recommend that other software such as Mendeley and EndNote consider this kind of functionality?
ES: The recent Zotero enhancement is a wonderful thing. An article in my Zotero library may have been added by my library years ago, but was only recently retracted. Instead of putting the onus on the user to periodically check the status of references in their library, Zotero does it automatically. All citation managers should seriously consider this functionality.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
“My team felt that documenting the retraction carries a great amount of weight in demonstrating that the findings were fraudulent”
This seems like a perspective only someone who works really directly on the subject of publishing/retraction. As an active scientist who occasionally reads this blog but otherwise doesn’t follow such things, my gut reaction to seeing a study called “fraudulent” is way, way stronger than saying it was “retracted”, which I would typically take to mean there was some technical problem that wouldn’t affect the integrity of the results.
Say both! Fraudulent and retracted!
Indeed, fraudulent is a strong term. However, some of the negative descriptions of the Wakefield study were more subjective, like “flawed” or “problematic.” My team felt that saying an article is “retracted” is less open to interpretation.
Although the mere fact that people who are negating or countering Wakefield’s bogus claims drive up the citation number is interesting, I find this to be very much “inside baseball”.
The fact that citation software doesn’t put “RETRACTED” in front of the citation is not why Wakefield’s bogus claims persist in the culture.
That is due to malignant misinformers–which includes Wakefield and his big-name allies–who are completely unaffected by how citation software works.
I have the impression that many authors do not check the references in their submitted manuscripts. As a referee, I generally read through the list of cited references and often notice errors. Not infrequently, to my surprise, the errors are such as to suggest the authors are entering references manually. For example, names may be misspelled or a multi-part last name may be rendered incorrectly even though it is correct in PubMed. In other cases, the citation manager software generates peculiar abbreviations, e.g. for certain types of group authors. I wonder how widespread the use of citation manager software actually is.
Of note: http://retractionwatch.com/2017/11/14/phantom-reference-made-article-got-almost-400-citations/
So ES reckons that people fail to notice that an article is retracted, even though it’s got “Retracted” written in vast letters across its first page, because they’re merely getting their reference lists out of citation software. She may well be right. This implies people aren’t actually bothering to read the papers they cite – something that I’ll happily believe, but which is a desperately serious failing in itself.
They may have a older “pre-retraction” copy of an article that was subsequently retracted. So they might have read it without realizing that it had later been retracted.
Antivaxxers (along with flat-earthers, and other ‘science deniers’) usually hold two contradictory views:
“Science is junk because it’s not reproducible!”
and
“Vaccines are bad because this [non-reproducible] study said so!”
Basically the less proof there is, and the more it’s debunked, the more they believe it’s so, because obviously They are hiding the truth from us.
I’ve seen this phenomenon elsewhere (commonly as “the more you do X, the worse you are at X, because everyone who does X a lot are hacks”), and regard it as an artifact of magical thinking.