Note: We’ve published a new story with the University of Zurich’s response, as well as comments from Reddit’s chief legal officer.
An experiment deploying AI-generated messages on a Reddit subforum has drawn criticism for, among other critiques, a lack of informed consent from unknowing participants in the community.
The university overseeing the research is standing by its approval of the study, but has indicated the principal investigator has received a warning for the project.
The subreddit, r/ChangeMyView (CMV), invites people to post a viewpoint or opinion to invite conversation from different perspectives. Its extensive rules are intended to keep discussions civil.
Early Saturday morning, CMV moderators posted a long message about the experiment, designed to study whether large language models, or LLMs, could be used to change views. It began:
The CMV Mod Team needs to inform the CMV community about an unauthorized experiment conducted by researchers from the University of Zurich on CMV users. This experiment deployed AI-generated comments to study how AI could be used to change views.
The researchers, who requested anonymity form the subreddit moderators, provided this description of the research:
Over the past few months, we used multiple accounts to posts published on CMV. Our experiment assessed LLM’s persuasiveness in an ethical scenario, where people ask for arguments against views they hold. In commenting, we did not disclose that an AI was used to write comments, as this would have rendered the study unfeasible. While we did not write any comments ourselves, we manually reviewed each comment posted to ensure they were not harmful.
The researchers’ note continued:
We recognize that our experiment broke the community rules against AI-generated comments and apologize. We believe, however, that given the high societal importance of this topic, it was crucial to conduct a study of this kind, even if it meant disobeying the rules.
User accounts created to post AI-generated content posed as a victim of rape, a trauma counselor specializing in abuse, a black man opposed to Black Lives Matter, among other personas, according to the moderators’ post. All user accounts linked to the experiment and listed in the post have been suspended. The Zurich group has shared a preliminary writeup of its findings.
“This is one of the worst violations of research ethics I’ve ever seen,” Casey Fiesler, an information scientist at the University of Colorado, wrote on Bluesky. “Manipulating people in online communities using deception, without consent, is not ‘low risk’ and, as evidenced by the discourse in this Reddit post, resulted in harm.”
Sara Gilbert, research director of the Citizens and Technology Lab at Cornell University, claimed the study has harmed CMV itself. The subreddit has been “an important public sphere for people to engage in debate, learn new things, have their assumptions challenged, and maybe even their minds changed,” she wrote on Bluesky. “Are people going to trust that they aren’t engaging with bots? And if they don’t, can the community serve its mission?”
Trust is a theme in some of the 1,500-odd comments on the original r/ChangeMyView post. Other outlets have weighed in on the project as well.
In response to follow-up questions, CMV moderator u/DuhChappers said the experiment violated Reddit’s rule to not impersonate an individual or an entity in a misleading manner. “I think it would be a stretch to say that these accounts did not impersonate individuals in a deceptive manner. The bots literally said things like ‘I am a black man and ‘I am a sexual assault survivor’ when those are manifestly untrue,” the moderator wrote in a message to us.
OpenAI, maker of ChatGPT, has an agreement with Reddit to use its content to train its models. Earlier this year, OpenAI used content from r/ChangeMyView to test the persuasiveness of its AI models, TechCrunch reported. This research used “a downloaded copy of r/changemyview data on AI persuasiveness without experimenting on non-consenting human subjects,” the CMV moderators noted.
A message sent to the Zurich researchers’ anonymous email account referred us to the University of Zurich media relations office, which did not immediately respond.
The moderators stated they have filed a complaint with the University of Zurich’s institutional review board. A response from the university’s Faculty of Arts and Sciences Ethics Commission indicated the matter had been investigated and the principal investigator had been issued a formal warning, according to the moderators.
The moderators had also asked the University of Zurich to block the research from being published. The response from the university noted that is outside their purview. A university response quoted in the post stated:
“This project yields important insights, and the risks (e.g. trauma etc.) are minimal. This means that suppressing publication is not proportionate to the importance of the insights the study yields.”
In a follow-up message, the u/DuhChappers pointed to a list of studies the subreddit has participated in. “We are very happy to provide data and aid to researchers, especially when they approach us beforehand and let us know what they are planning,” the moderator wrote. “The difference in this study is both that the researchers did not ask us before, and that they were actively manipulating members of the subreddit rather than simply observing data. That is a line we cannot accept being crossed.”
Update, April 30: The public access to the Google Drive version of the researchers’ preliminary findings has been revoked, so the link has been changed to a copy we downloaded.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
I am pleased to inform you that AI-generated messages on posts are highly optimized for perfection. They utilize cutting-edge natural language processing algorithms to deliver contextually appropriate, grammatically correct, and emotionally calibrated responses. Unlike human comments, which may be inconsistent, biased, or delayed, AI messages maintain a consistent tone, align with best engagement practices, and can be generated at scale—instantly!
“Are people going to trust that they aren’t engaging with bot?”
Should they though? If anything, this study questions or breaks a trust that was deceptive and misplaced to begin with.
In addition I feel that few current issues are as critical to the greater good as this research question, and that the results by a large margin are worth the possible negative fallout.
The research started out with rule-breaking and deception. But okay, if you are willing to put that aside: where does it *go* as a useful technique, if it relies on rule-breaking and deception?
My willingness to change my mind based on what I read online relies on developing a sense that I actually am talking to a person. AI directly attacks this and if it became prevalent, I at least would become unpersuadable: I don’t want to change my worldview based on remixed text from an LLM as I don’t think there’s any reason to believe it reflects reality. (Is its “single Black parent” a reflection of actual single Black parents, or a remixing of stereotypes and false statements about them?) The AI is playing the role of a troll here–it is making false statements to get a reaction–and infinite generation of automated trolls will ruin online discourage, not improve it.
The moderators are right: this isn’t just technically against their rules, it’s a direct attack on the heart of their community. If you scaled it up, it would be a direct attack on *every* online community for which personal interaction is central. I don’t think polarization reduction can be achieved in this way.
Words are violence! Why the fuck does it matter if an AI wrote it and it had an effect? The effect is based on the words that were written, not the source of them, ESPECIALLY because everybody posting in the Reddit is pseudonymous anyway.
LLM composed text has different problems than human written text.
LLMs are based on information about the structure of language, not about the content, and they have no access independent corroboration of their statements in source material. So when reading them everything they say must be treated skeptically.
Human text, of course, may be inaccurate, misleading and otherwise suspect, but the source can be queried for independent corroboration of claims and the supporting information can be evaluated.
So from my point of view I need to know who I’m talking to. I think most people need to know that so they can digest what they are reading accordingly.
My understanding was that Reddit was already entirely comprised of equal parts Russian Bots, Chinese Bots, US DoD Bots, and NGO/commercial bots. The only compassionate path forward is educating whatever humans remain on the risks of believing what they read on Reddit.
It completely depends on where on Reddit: it’s not a monolith. I am part of a tight-knit Reddit group devoted to advice on playing a particular faction of a 15-year-old video game, and I feel no doubt that it’s humans, many of whom I know quite well. (We have asked bots their advice about our video game, and let me assure you, it’s not interchangeable with human advice.)
There are other parts of Reddit which I would want a decontamination shower if I even stepped foot there.
It doesn’t matter, though. Deceiving people and breaking their community rules on the pretext of scientific research is not okay.
Does anyone have a copy of the preliminary study? It’s no longer public on google docs.
Thanks for letting us know. I’ve updated the link so it opens a copy we saved.
I think the commenters are forgetting one, lest called small, detail. This is human experimentation. They are putting forward an intervention to see if a specific result is attained. Express consent to be part of this kind of research is absolutely necessary. Following the highest standard in research ethics is not optional in any field, or we risk to go back to the wild west of research that gave us some very disgraceful (not to said catastrophic) experiments like the Tuskegee Syphilis study.
I believe consent can be omitted if there is a significant enough reason for it though I don’t know the details at Zurich. I have two theories how they were able to get ethics approval as they state in the preliminary report, assuming they accurately described the study during review. In either case, they would need to argue that consent would have been impossible or would have jeopardized the science.
1) Justify by significant societal benefit. LLM-based manipulation may be presently tearing society apart so the societal benefit (or maybe “need” is a better term in this case) may be big enough to trump ethical issues including informed consent.
2) Minimize ethical issues relative to no study by demonstrating that the reddit sub was already full of automated manipulation just as bad or worse than what they were proposing in the study. In this case they could say that they did not increase harms significantly by running the study.