Why “good PhD students are worth gold!” A grad student finds an error

Researchers in the Netherlands have retracted and replaced a 2015 paper on attention after discovering a coding error that reversed their finding. 

Initially titled “Effects of Transcranial Direct Current Stimulation over Left Dorsolateral pFC on the Attentional Blink Depend on Individual Baseline Performance,” the paper appeared in the Journal of Clinical Neuroscience and was written by Heleen A. Slagter, an associate professor of psychology at VU University in Amsterdam, and Raquel E. London, who is currently a post-doc at Ghent University. It has been cited 19 times, according to Clarivate Analytics’ Web of Science.

But while trying to replicate the findings, Slagter and a then-PhD student of hers, Leon Reteig, found a critical mistake in a statistical method first proposed in a 1986 paper. Slagter told us: 

In this study, to determine the relationship between baseline task performance and the effect of tDCS and to ensure that this was not driven by regression to the mean, we used an excel sheet from the MRC Brain and Cognitive Sciences unit of the University of Cambridge, that implements an unbiased method by Mrytek and Foerster (1986) for assessing the influence of baseline on change from baseline, freely shared online. See http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/rxxy_correction

Reteig wanted to use this same correction and discovered an error in its implementation in this Excel file. This error was very difficult to find as you had to unlock the excel file for this, to show the hidden code, and then double check the formulas. 

The Excel file had been posted to the University of Cambridge website in 2011 by Peter Watson, who confirmed the error and corrected it in a new file. 

The mistake, Slagter said: 

affected one of the main conclusions of our 2015 paper, as when correctly estimated, we no longer observed an effect of transcranial direct current stimulation over left dorsolateral prefrontal cortex on the attentional blink as a function of individual baseline performance. However, the two other main findings were unaffected (as they did not depend on this analysis). As the affected main conclusion was also expressed in the title of the paper, in consultation with the journal we decided to retract the paper and publish the paper with this main finding corrected. 

As the retraction notice states: 

While attempting to replicate and extend the original work, an error was discovered in the routine that implemented the analysis used to assess whether the relationship between an independent variable and a dependent variable is driven by regression to the mean (Mrytek & Foerster, 1986). After the error was corrected and the analysis was rerun, it was found that one of the main results—an influence of baseline level of performance on anodal tDCS induced change in the attentional blink, presented in Figure 3—no longer held. Given that this finding was also expressed in the title of the article, the authors requested the article to be retracted. The complete set of results from this study, now with the corrected analysis, will be published as London R. E., & Slagter, H. A. “No effect of transcranial direct current stimulation over left dorsolateral prefrontal cortex on temporal attention” in the Journal of Cognitive Neuroscience, https://doi.org/10.1162/jocn_a_01679.

For Slagter, the lessons of the experience are clear: 

always double check code, even code used by others in the past. And good PhD students are worth gold!

Indeed.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].