Last month, we reported on a retraction in Judgment and Decision Making that said “problems were discovered with the data.” At the time, corresponding author Wen-Bin Chiou, of National Sun Yat-sen University in Taiwan, told us that a research assistant who had since left the lab hadn’t kept questionnaires used in the research, making replication impossible.
But it turns out that wasn’t the whole story of “Shame for money: Shame enhances the incentive value of economic resources,” to put it charitably. We’ve now heard from people familiar with the case and can provide a fuller account.
The problems with the paper, according to our source,
began with the large size of the effect of the emotion manipulation on the coin sizes, and the high correlation between the coin-size effects and the dependent variables.
The originally published version of the paper was actually a revision of a previously rejected manuscript that answered reviewers’ concerns. But
The current version answered those questions so well as to be implausible…
That, however, didn’t jump out at the time, so the paper was published. Once it was, at least one reader noticed that the data were excessively similar across conditions, and — similar to the cases of Lawrence Sanna, Dirk Smeesters, and Diederik Stapel,
data across supposedly independent samples appear too similar to have arisen from random samples.
At this point, Chiou was given the chance to retract the paper, which he did.
As to whether the data had disappeared, as he told us, it turns out Chiou said that he had actually reviewed the data by hand in response to questions about why the coin-size numbers in the data were all integers, even though they were supposed to be averages of four integers. Those concerns weren’t the reason for the retraction. But they suggest Chiou wasn’t being completely forthright with us, so we wanted to set the record straight.
Finally, worth noting: Judgment and Decision Making is one of the few behavioral science journals that requires
posting of data. The journal instituted that policy a few years ago, and this clearly led to discovery of this problem. A good lesson for journals reluctant to take such a step, we think.
Update: See psychology sleuth Uri Simonsohn describe how these problems were uncovered.