Science has issued an expression of concern for a highly publicized study looking into whether conversations with AI chatbots could convince conspiracy theorists to abandon their beliefs. The move came after the authors of the paper found inconsistencies in their dataset, but a reanalysis shows the findings still stand, they say.
The September 2024 article found conversing with an AI chatbot called DebunkBot reduced people’s belief in a particular conspiracy theory by an average of 20%. The research was featured in news stories in The New York Times, Washington Post and The Atlantic.
This February, the authors — Thomas Costello of Carnegie Mellon University in Pittsburgh, psychologist Gordon Pennycook of Cornell University in New York and cognitive scientist David Rand at the Massachusetts Institute of Technology — won the Newcomb Cleveland Prize from the American Association for the Advancement of Science, which publishes Science, for the work. It has been cited 192 times, according to Clarivate’s Web of Science.
According to the notice published today, the authors learned of issues with the public dataset that “made it challenging to reproduce some of the specific values reported in the manuscript.” After investigating, the authors also discovered “inconsistencies in the application of screening criteria between the manuscript and published analysis pipeline.”
The researchers reported the issue to Science and provided their raw datasets along with updated results, which the journal is now evaluating, the notice says. The reanalysis “produces results that match those in the original article in direction, statistical significance, and substantive size.”
“I’m very grateful that these issues were surfaced and to have a chance to correct some methodological and supporting details of the paper in the public record,” Costello told Retraction Watch.
Costello declined to provide additional details while the reanalysis is under review. “None of the findings or conclusions will change as a result of this process and the public record will be more accurate. So, a good outcome,” he said.
Meagan Phelan, communications director of Science journals, told us the trio informed the journal of the issues last month.
Shortly after the paper was published, a PubPeer user wrote in a comment the paper “makes a number of claims with little information on how to assess their credibility.” But subsequent comments pointed out the article addressed many of the user’s questions. Phelan told us the inconsistencies that led to the expression of concern were unrelated to the comment.
Political scientist Brendan Nyhan, who commented on the paper for The New York Times when it was first published, told us he and his students have reproduced the paper’s results in two independent experiments at Dartmouth College in New Hampshire.
The expression of concern “shows the importance of journals doing full replications of article results from author-provided data and code prior to acceptance and publication,” Nyhan said. “It’s not perfect but it could help to prevent a lot of problems.”
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].