A paper about the effects of generative AI use on confidence in work tasks is under investigation after critics raised questions about the study design, data analysis and ethics approval for the research.
The study, published in April in Technology, Mind, and Behavior, included 1,923 adults recruited online from the United States and Canada to perform a battery of work-like tasks assisted by AI. It garnered a press release from the American Psychological Association, which publishes the journal, and coverage in Time and Futurism.
Sandra Grinschgl of the University of Bern in Switzerland, who studies technology-driven cognitive offloading, got an alert about the study shortly after it came out. She told Retraction Watch she was initially puzzled by the vague descriptions of data collection from online participants in the study. When she looked closer at one of the bar charts, she noticed the lengths of the bars didn’t match the actual value of the labels.
To dig deeper, she enlisted two colleagues at the University of Bern who are well-versed in scientific sleuthing: Ian Hussey, who helped create the INSPECT-SR reproducibility checklist for research papers, and psychologist Malte Elson, who has been involved in assessing the trustworthiness of published research for over a decade and whose work previously led to the retraction of a high-profile paper about the psychological impact of first-person shooter video games.
“Taking a step back, we realised that some more fundamental claims about the study design and execution seem implausible without further information or access to study materials,” Elson told us by email.
The researchers shared their concerns on PubPeer, LinkedIn and Bluesky, where other researchers “started to look into it and found additional things that we didn’t discover before,” Grinschgl said.
The Bern researchers also alerted Richard N. Landers, the editor-in-chief of Technology, Mind, and Behavior, and the article’s action editor Christopher Ferguson. Ferguson told us an investigation is under way, but declined to comment further until the inquiry is complete.
The study’s sole author is Sarah Baldeo, founder and CEO of ID Quotient Advisory Group, a firm that provides “neuroscience-enabled IT & AI consulting,” according to its website. She is also a Ph.D. candidate in AI and neuroscience at Middlesex University in England, though the work for the study predates her time in the program, Allison Phipps, ID Quotient’s chief of staff, told us by email.
The stated mission of ID Quotient is to “ensure technology works in harmony with the human brain” with custom AI tools, strategy services and workshops for organizations. The company’s website previously included a client testimonial from the APA. When asked what services ID Quotient provided the association, Phipps clarified that Baldeo “has previously delivered keynote presentations for certain APA-affiliated divisions or society groups.” As of June 1, the client quote attributed to APA was no longer listed on the ID Quotient website.
Baldeo has responded to some critiques of the paper on PubPeer and LinkedIn, and she and Phipps clarified factual details via email, but Baldeo’s executive assistant declined our interview request on her behalf, and Phipps did not respond to follow-up requests.
Grinschgl, Hussey and Elson put the article through the INSPECT-SR checklist and found several discrepancies between the descriptive statistics and the actual results. “The results as reported appear to be mathematically impossible,” Grinschgl wrote in her initial PubPeer comment, which also scrutinized the paper’s methods, such as how responses were recorded as users interacted with AI on the task battery. “We are not aware of research platforms of this sophistication,” she added.
Around 600 participants were reportedly senior leadership or executives, a population that is “notoriously difficult to recruit,” Elson noted in his email.
Grinschgl’s team also observed that the paper was missing a statement of approval from an institutional review board.
Baldeo has since presented proof that the study was approved by a Canadian research ethics board, Landers told us. He said the editors at the journal were reviewing this as part of their investigation, and described the initial lack of an approval statement as an oversight. “I’ve revised our instructions to associate editors and updated our process to prevent it from happening again,” Landers wrote in an email to Retraction Watch.
While the paper’s experiments measured confidence and reliance on AI, the article’s title mentions executive function and cognitive harm, which Baldeo didn’t evaluate in the study, Hussey pointed out on PubPeer. Others questioned mismatches between descriptions in figures versus in the article text, such as the x-axis label on a bar chart of self-reported AI reliance that didn’t correspond to any of the items in the article’s description of self-reported measures. Commenters also questioned why Likert-style responses were represented with percentages on the bar chart.
Sleuths surfaced a nonexistent citation attributed to University of Irvine professor Craig Stark, who confirmed via email that no such paper exists. Baldeo later commented on PubPeer offering another paper with the same first author and a different title as the correct reference. She also characterized the mismatch between lengths and percentages on the bar chart as a rendering error, and Phipps told us that she has now provided the journal a corrected figure.
In the comments of a LinkedIn post about her work, Baldeo claimed she had fMRI data for a third of the study participants. PubPeer commenters questioned the feasibility of collecting fMRI data from hundreds of people across the United States and Canada. In the comments of another post, she stated that pre- and post-study fMRI data was provided by participants themselves.
This scenario “does not seem any more plausible” than Baldeo conducting the fMRI scans, Elson said. Baldeo countered on PubPeer that the scans were performed separately and are thus irrelevant to critiques of her article.
“The article was framed and reviewed as a descriptive, exploratory, nonclinical behavioral study. It does not claim causal effects, cognitive decline, neural change, or clinical impairment,” Baldeo wrote in another PubPeer comment. In one of the LinkedIn posts about her work, Baldeo wrote that “GenAI use is NOT destroying your brain. As the lead researcher for the largest GenAI & Brain study in the world I can confidently and firmly state this.”
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
“It garnered a press release from the American Psychological Association”
Disappointed that you couldn’t weave “delved” into the post as well.