Can we do math unconsciously? Replicators of a prominent 2012 study have some doubts

In 2012, news media were abuzz with a new finding from PNAS: Authors based in Israel had found evidence that our brains can unconsciously process more than we thought — including basic math and reading.  In other words, the authors claimed people could read and do math without even knowing what they were doing.

With such a major development in the field of consciousness research, other groups quickly got to work trying to replicate the findings. Those efforts have taken some twists and turns — including a recent retraction of a replication paper that was, itself, not reproducible (which is not something we see every day). But overall, five years after the initial, remarkable result, the replication efforts are calling it into question.

According to Pieter Moors at KU Leuven, a researcher in this field:

…the most parsimonious conclusion is that no strong evidence exists for unconscious arithmetic as reported by [the 2012 PNAS paper].

“Reading and doing arithmetic nonconsciously” has been cited 68 times since it appeared in 2012, according to Clarivate Analytics’ Web of Science, formerly part of Thomson Reuters.

Here’s how the New Scientist described one of the experiments:

The technique works by presenting a volunteer’s left eye with a stimulus – a mathematical sum, say – for a short period of time, while bombarding the right eye with rapidly changing colourful shapes. The volunteer’s awareness is dominated by what the right eye sees, so they remain unconscious of what is presented to the left eye.

In the team’s first experiment, a three-part calculation was flashed to the left eye. This was immediately followed by one number being presented to both eyes, which the volunteer had to say as fast as possible. When the number was the same as the answer to the sum, people were quicker to announce it, suggesting that they had subconsciously worked out the answer, and primed themselves with that number.

Moors told us he was initially blown away by the 2012 finding:

This finding was also particularly controversial because it implied much more extensive unconscious processing than implied by previous research on the topic…

So Moors was pleased to see when the authors of a 2016 paper tried to replicate the findings — but was puzzled when it showed different results. Specifically, the 2016 paper showed the brain could process math equations unconsciously — but not in the same way the 2012 paper found:

The study of Karpinski et al. was a welcome addition to the literature exactly because it provided an exact replication of the study of Sklar et al. (be it in a different populations of students). They only replicated the arithmetic part of the study, but actually observed the opposite effect compared to Sklar et al. That is, Sklar et al. had observed unconscious arithmetic for subtraction but not addition, and Karpinski et al. for addition but not subtraction.

Since the findings were so different, Moors and his colleague Guido Hesselmann at Charité-Universitätsmedizin in Germany asked for both datasets, in order to analyze (and possibly combine) them. That’s when they noticed a problem:

Both authors shared their data sets without any problems, and when we started to reanalyze the Karpinski data set, we noticed that there was a potential problem with the dependent variable that was used to analyze the data. After some digging, and emailing back and forth with the authors, we eventually discovered that this problem was caused by the fact that the dependent variable was incorrectly calculcated for a certain condition (not all rows were selected to calculate the particular variable). When the data were reanalyzed without this calculation error, it appeared that the replication with different results turned into a failure to replicate (i.e., no unconscious arithmetic effects were observed anymore).

Indeed, once Andrew Karpinksi at Temple University realized he’d made a mistake in processing the data, he contacted the European Journal of Social Psychology to retract his paper. Specifically, he told the journal that although his paper initially appeared to reinforce the concept behind the 2012 high-profile paper, once he redid the calculations, he was unable to replicate the findings at all. Here’s the letter he sent to the journal, which Karpinski forwarded to us:

It has come to our attention that there was a substantial error in our calculations.  This error is not an errata in our published results, but one that results in a fundamental change in our results in conclusions.

Specifically, we calculated mean response times using the number of correctly pronounced trials as the denominator, while we should have used the number of trials after eliminating those that were pronounced incorrectly and those that were greater than three standard deviations above the participant’s mean.

After correcting for this mistake, our results do not show a significant priming effect for addition equations, t(93)=.106, p=.916, d=.01, nor subtraction equations, t(93)=.230, p=.818, d=.02. Thus, it is incorrect for us to claim that our results (partially) replicate those of Sklar et al. (2012); instead, our results are a failure to replicate those of Sklar et al. (2012).  We find no evidence of unconscious arithmetic processing in our data.

Given this fundamental change in our conclusions, we believe that we must retract our article.

Furthermore, I (Andy Karpinski) take full credit for the mistake in our calculations.  I (Andy Karpinski) wrote the template that reduced the raw data to cell means, and it is I alone that is fully responsible for the mistakes in this manuscript.  None of the other co-others of this paper were involved in the construction of the template that reduced the raw data to cell means.

That paper — “Unconscious arithmetic processing: A direct replication” — has now been retracted. The notice echoes Karpinksi’s statement, and concludes:

After correcting this error, the authors find no evidence of unconscious arithmetic processing in their data.

Karpinksi added that he has submitted a revised version of the manuscript to the journal, which is now under review.

Meanwhile, Moors and Hesselmann have also attempted to replicate the 2012 findings, as they report in a recent paper in Psychonomic Bulletin & Review. As Moors told us:

Coming back to our reanalysis of the Sklar et al. data set, we found that the statistical evidence for the results they reported was much weaker than we thought…When we take the results of all our reanalyses together, we concluded that the evidence for unconscious arithmetic is not as convincing as originally reported…

Moors added that another paper has failed to replicate the reading portion of the 2012 paper, and a different researcher has criticized the 2012 analysis on different grounds.

We contacted the corresponding author of the 2012 paper, Ran R. Hassin of Hebrew University in Jerusalem. He told us:

My colleagues and I have been paying very careful attention to the recent replication attempts concerning our 2012 PNAS paper, and to novel attempts at analyzing our data.  My lab is working closely with one of the labs, we gave our scripts and full data to the others, and I look forward to reading the details of the upcoming papers.  We are happy to encourage and facilitate every effort at replication/analyses and are pleased at such attempts.  This does not mean, of course, that we completely agree with the approaches, methodologies, and the arguments. We continue to stay engaged with this work and are ready to potentially adjust our conclusions and theorizing as more data are collected (by our lab and other labs).

Update June 4, 2018 16:00 UTC: We’ve heard from PNAS study author Ran Hassin, who told us researchers have republished the retracted replication.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.