‘I have zero complaints about the process’: Post-publication analysis earns perception paper a flag

A journal has issued an expression of concern for a nine year old paper, which purported to find that people associate morality with brightness (that’s light, not smarts), after a data sleuth found problems with the results. 

The article, “Is It Light or Dark? Recalling Moral Behavior Changes Perception of Brightness,” appeared in Psychological Science in 2012 and was written by a group of marketing researchers at the Winston-Salem State University, in North Carolina, the University of Kansas and the University of Arizona. 

Aaron Charlton, a marketing researcher at Illinois State University who’s involved in replication efforts in his field, told us that he decided to take a closer look at the data in the paper, which he noted had been the subject of two previous attempts to replicate the key findings, after seeing this post on PubPeer. 

Two of the authors, Promothesh Chatterjee and Jayati Sinha, lost a 2013 paper in Marketing Letters for “unexplained anomalies” and “coding errors” in the data. And Sinha also received an somewhat controversial expression of concern for a 2019 article in the Journal of Marketing.  

Using the GRIM test — short for granularity-related inconsistency of means — from Nicholas Brown and James Heathers, Charlton saw that some of the results the authors reported appeared to be mathematically impossible. 

Or, as one commenter on Twitter, Quentin André, posted about the expression of concern: 

I’m tempted to be sarcastic here, but I’ll try to be constructive instead: If someone posted a picture of 12ft tall human, 99% of people would conclude… that the picture is photoshopped. A claim of 12ft human is extraordinary, and requires extraordinary evidence. Why don’t we treat d > 1 for subtle psychological effects in the same way?

Charlton contacted the journal, which did its own investigation and concluded that he was correct. 

According to the journal: 

The Expression of Concern is prompted by the observation that some of the mean values reported in the article are mathematically impossible, given the manner in which the data are reported to have been collected. The authors of the article attempted to locate the original data in an effort to resolve the errors, but they were unsuccessful. Because the errors cannot be resolved, we decided to issue an Expression of Concern about the confidence that can be held in the reported results. The corresponding author on the article, Promothesh Chatterjee, was invited to contribute to the Expression of Concern as a coauthor but declined the invitation. …

The errors in the data reported in the article were brought to the attention of the Editor in Chief by Aaron Charlton, who conducted a statistical check known as a test for granularity-related inconsistency of means (GRIM; Brown & Heathers, 2017). The GRIM test assesses whether reported statistics could have been created from a data set. In the present case, the question was whether the reported mean values are mathematically possible given (a) the number of participants in the studies and (b) the reported scale of measurement of the variables. Charlton observed that on the basis of his calculations, five of the 10 means reported in the article were mathematically inconsistent.

On receipt of Charlton’s message of concern, the Editor in Chief recruited the assistance of the coauthor of this statement, Gregory Francis, one of the Statistical Advisers for Psychological Science. Francis conducted an independent analysis of the values reported in the article. He concluded that given the reported sample sizes and scale of measurement, there are two means reported in the article that are mathematically impossible.

The notice concludes: 

In summary, if the scale of measurement involves integers, as described in the article, then the GRIM test reveals errors in some of the reported mean values. Either the reported numbers are incorrect, the reported sample sizes are incorrect, or the description of measurement is incorrect. The problems could reflect simple mistakes in reporting (e.g., typos or missing data), but because the original data for this article cannot be located, the errors cannot be corrected. With this in mind, the Editor in Chief decided not to change the official publication record of the article through a Corrigendum. Instead, we issue this Expression of Concern and note that the errors undermine confidence in these data and the conclusions drawn from them.

Banerjee now appears to be affiliated with the Indian Institute of Management, in Kozhikode. Sinhal, who holds the Macy’s Retailing Professorship at Florida International University, did not respond to a request for comment. 

The article has been cited 38 times, according to Clarivate Analytics’ Web of Science. 

Recceiving credit for sleuthing in a notice is still unusual. On his blog, Charlton wrote that: 

I was very pleased with how Patricia Bauer, the editor of Psych Science handled everything. She was super appreciative and kind. And Greg Francis did an excellent job on the analysis and writeup I thought. I have zero complaints about the process. They also handled everything very quickly. 

Charlton added in an email: 

I would like to say that I don’t work alone. I have help from anonymous individual(s) who don’t feel comfortable going public because of negative career and personal ramifications. It’s not that they’re afraid–it’s just that there appears to be no upside to going public. And they’ve been attacked for some of their efforts. Science won’t truly be self-correcting until we make it safe for people to speak up when they’ve discovered a problem without fear of negative repercussions. 

Like Retraction Watch? You can make a one-time tax-deductible contribution by PayPal or by Square, or a monthly tax-deductible donation by Paypal to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].