In the midst of the holiday season, it’s a pleasure to be able to share the story of a scientist doing the right thing at significant professional cost — especially a researcher in psychology, a field that has been battered lately by scandal.
Sometime after publishing two papers — one in Developmental Science and another in the Journal of Personality and Social Psychology – Yale’s Laurie Santos and her students realized there were problems with their data. We’ll let Santos — who made sure to respond to our request for comment immediately, in the midst of holiday travel, so that we had all the details and could help get the word out — tell the story:
My students and I were trying to replicate the results of Experiments 1-3 that we published in Mahajan et al. (2011) JPSP with a larger sample of monkeys (Note: since we know replication tends to be a problem in primate studies in particular due to subject access, it is our lab’s policy to regularly try to replicate our own effects with new samples at our field site, just to be sure the effects we report hold in different samples and to serve as baseline conditions for follow-up studies).
In this case, we were also trying to extend the sample we had initially tested to look for individual differences in the magnitude of the “ingroup bias” we reported in the JPSP paper. Our JPSP paper observed that when monkeys were presented with photographs of ingroup and outgroup members, they looked significantly longer at the outgroup faces than the ingroup faces (see effects in Experiments 1-3). When we tried to replicate this pattern in a larger sample, we didn’t observe the original effect. Instead, we saw that monkeys didn’t show any consistent overall difference in looking across the ingroup and outgroup faces. We thought we could have failed to replicate the effect reported in the JPSP paper for a number of uninteresting reasons (e.g., several other research groups had recently started using photographs at the field site, and therefore monkeys may have habituated to pictures overall, etc) but just to be on the safe side, we decided to go back to the initial videos from Mahajan et al. and have new coders recode them all from scratch.
This was when detected the problem with Neha Mahajan’s (the first author) coding of the original datasets. We then quickly realized that Neha had probably used the same coding techniques in other studies she worked on and thus that similar problems might be present in other datasets that Neha had coded in the lab. So we went back and checked the original coding for all other studies Neha had coded as well. This was how we caught the second set of errors in the now-retracted Developmental Science paper you wrote to me about initially.
Since both of the coding problems resulted in results that no longer existed, we thought it was our responsibility to report the situation to the university. They did indeed perform an investigation, but found that Neha was not guilty of any misconduct or negligence. It seems it was just human error. (Sucky annoying human error that resulted in the retraction of two papers, but human error nonetheless).
The retraction notices include a good amount of detail from that narrative. Here’s the one from Developmental Science:
Neha Mahajan, Jennifer L. Barnes, Marissa Blanco, Laurie R. Santos “Enumeration of objects and substances in non-human primates: experiments with brown lemurs (Eulemurfulvus)” Developmental Science, Volume 12, Issue 6, pages 920–928, 2009
The above article, published online on 18 May 2009 in Wiley Online Library (wileyonlinelibrary.com), has been retracted by agreement between the authors, the journal Editors in Chief, Michelle De Haan and Charles Nelson and John Wiley & Sons Limited.
The authors determined that the looking time coding performed by the first author, N. Mahajan, was inaccurate and did not reflect the looking times found by other trained coders. A reanalysis of the data reported in this paper failed to verify the reported effects, and thus the authors have requested that the publication be retracted.
Ms. Mahajan takes sole responsibility for the inaccurate coding. An investigation found that the inaccurate coding was not caused by intentional, knowing, reckless, or grossly negligent action by Ms. Mahajan. All authors of the original publication joined in the request for retraction.
The now-retracted study has been cited seven times, according to Thomson Scientific’s Web of Knowledge. And here’s the retraction in the Journal of Personality and Social Psychology, which will appear in the January issue:
The following article from the March 2011 issue is being retracted: Mahajan, N., Martinez, M., Gutierrez, N. L., Diesendruck, G., Banaji, M., & Santos, L. R. (2011). The evolution of intergroup bias: Perceptions and attitudes in rhesus macaques. Journal of Personality and Social Psychology, 100, 387– 405. doi: 10.1037/a0022459
The retraction is at the request of the authors. This article reported two independent sets of effects concerning monkeys’ intergroup behavior: (a) that there are differences in monkeys’ vigilance toward ingroup and outgroup members (Experiments 1–5) and (b) that monkeys show implicit associations toward ingroup and outgroup members (Experiments 6 and 7). After lab members were unable to replicate the first set of published effects for new research purposes, the authors determined that the looking time coding performed by one of the coauthors, N. Mahajan, was inaccurate and did not reflect the looking times found by other trained coders. Ms. Mahajan had been the sole coder for most of the studies, resulting in the reporting of inaccurate data for Experiments 1–5. The coding performed by the other coauthors was accurate. A small subset of the studies were doublecoded for reliability, but this coding was not used in the overall analyses reported in the article.
A full recoding of the data from Experiments 6 and 7 verified the conclusions from the second set of findings reported in this article. The results of Experiments 6 and 7 will be resubmitted for publication as a separate article.
Ms. Mahajan takes sole responsibility for the inaccurate coding. A formal investigation conducted by Yale University found that the inaccurate coding was not caused by intentional, knowing, reckless, or grossly negligent action by Ms. Mahajan.
All authors of the original article joined in the request for retraction.
That study has been cited 22 times.
Santos reflected on the experience:
Having to retract papers is a scientist’s worst nightmare. Especially in the current climate in psychology right now (e.g., Hausergate, Stapelgate, etc…), this is pretty much the most awful thing that could happen to a PI. But I also hope that this awful situation can– at least in some sense– serve as a positive example of correcting the scientific record. We would have never caught the coding error without replicating the initial JPSP effects (which particularly given the subject access that plagues primate cognition work is something that more and more scientists need to do). And as soon as we found the problems, we immediately went back and checked all the other datasets too. I’m obviously embarrassed that we didn’t catch all this earlier, but I’m still glad that we caught it when we did.
The fact that Santos mentioned another Ivy League psychology researcher who studied monkeys is a reminder of the stark contrast between the straightforward way she approached these problems and the way that Marc Hauser — with whom she studied at Harvard — chose to. Kudos to Santos and her colleagues.
Hat tip: Rolf Degen