Where did I park my car? Psychological Science retracts working memory study

In what might be considered a model for how retraction notices should look, Psychological Science has retracted a 2008 paper, “Gaining control: Executive training and far transfer of the ability to resolve interference.” According to the notice — which includes two tables:

This article has been retracted by both of the authors. During further extension of this work, it was discovered that a mistake had been made in the programming of the working memory training tasks used in this study. This error resulted in the presentation of repeated trial sequences, which were then practiced repeatedly over the 8-day training period. This practice likely resulted in learning of the sequences, rather than training of interference control as we had originally inferred. The first author takes responsibility for this error, and both authors regret the publication of invalid results. In a separate replication (N = 16) of the working memory training condition that used randomized, novel sequences for each of the 8 training days, we were not able to reproduce the main findings of the original report. Table 1 presents inferential statistics comparing performance in the Day 1 and Day 8 training sessions, in the originally published data and the data from the attempted replication. As the table shows, no significant training effect (i.e., reduced interference resolution) was observed for either type of stimulus (faces, letters) when the correctly randomized trial sequence was used. Table 2 presents inferential statistics comparing performance on the transfer tasks before and after training, in the previously published data and the new data. As the table shows, no significant effect of training on performance in any of the three transfer tasks was found in the new data set.

We liked the fact that the authors gave such great detail about what had led to the retraction, and spelled out what happened when they tried the experiments again.

The original study was been cited 19 times, according to Thomson Scientific’s Web of Knowledge. It has been completely removed from the Psychological Science site, but a preprint is available here.

We asked Vaughan Bell, of the terrific neuroscience and psychology blog Mind Hacks, to explain the study to us. Read on for how the work could apply to finding your car in a garage, or tango lessons:

The study is not the easiest to grasp because its a fairly technical look at the ‘mechanics’ of how the executive system is used to resolve cognitive conflict. The ‘executive system’ or ‘executive function’ is the cognitive system usually associated with the pre-frontal cortex which has the job of managing other cognitive resources. It’s literally thought of as the high-level management of the cognitive system / brain.

As we get better at tasks they become more automatic and hence require less executive input. However, this process tends to be quite task-specific and there is a big interest in whether it is possible to see improvement that will generalise to other tasks (this has been fuelled partly by the hype surrounding ‘neuroplasticity’ and partly by the hype surround ‘brain training’ – needless to say, there is some genuinely interesting science behind the hype, but undoubtedly it has received some reflected glory from the popular interest).

The executive system is usually thought of as operating by the deployment of attention, which can be both unconscious (usually conceptualised as neural or cognitive ‘effort’)  or conscious (which can be either as self-directed concentration or by something ‘grabbing’ our attention). This attention can be employed to resolve conflicts caused by the inappropriate activation of automatic / well practised cognitive process. The example they give at the beginning of the paper is very good:

While searching for your parked car, you find yourself walking toward the spot where you parked yesterday or the day before, rather than where you parked today. To avoid such errors, we typically slow down, and employ deliberate strategies in an effort to resolve this interference from prior experience, and find today’s parking spot. That is, we invoke “cognitive control.”

In fact, this interference can apply for many types of memory tasks. Imagine you are given surnames and you have to remember first names, but on separate lists you have both a Mary Smith and a Marie Smith. Even though a name might jump to mind you have to concentrate and think to work out which name appeared on the list you are currently working on.

The study was attempting to show that practice using cognitive control to resolve conflicts in working memory would generalise to other tasks that haven’t been practised. In other words, if it worked, it would show that we could generally improve our ability to reduce a large number of mental errors. A big deal.

Previously the team had demonstrated that there is cross-task fatigue between a high-interference working memory task (that requires lots of cognitive control to prevent errors) and the ability to resolve interference in verb generation and episodic memory tasks. This suggests that the same cognitive control mechanism is involved in both.

Hence their idea was that they could give lots of regular practice on high-interference tasks to ‘train up’ cognitive control and afterwards see if this ability to resolve errors generalised to non-practice tasks.

So they gave the participants some pre-training tests in verb generation, word recognition and paired-associates (paired-associates is memory for word pairs, like the names example given above), then gave them eight days of working memory practice in either in a high interference, low interference or low interference and low memory load (‘easier to remember’) versions. After two weeks of daily practice they tested them on the pre-training tasks again to see if the training had made a difference.

They found that the participants in the high interference condition got better at the training tasks and improved in the final tests, suggesting cognitive control had improved and was generalised to new tasks.

The error was that the presentation of stimuli in the training tasks was not properly randomised and so their improvement was not in cognitive control but simply in learning the sequences (probably through recognition). If this isn’t clear, think of it like this. If I want to learn to tango, I can more easily learn a specific routine than the ability to freestyle it, because the routine just involves learning recognition and response rather than the skill of coordinating the underlying moves of tango. I haven’t actually learnt tango although my limited routine has improved. To learn tango, I need to practice all the moves out-of-sequence.

In their follow-up analysis when they properly randomised the stimuli, there was no improvement in the training tasks and no-improvement in the post-test evaluation. In words, the improvement was just them repeating the learnt sequences rather than a genuine improvement in general cognitive control.

It’s like if my tango teacher forgot that the training had regular certain sequences and that the final exam contained them too. She’d think I was a better tango dancer when it really wasn’t the case.

We asked Robert Kail, the editor of the journal, how the errors came to his attention. He said it was confidential. We also wanted to know whether it was typical for the journal to remove the original study entirely, rather than leave it and mark it as retracted:

This is the first retraction we’ve done in several years, so there really isn’t any “typical.”

He also said it was hard to tell whether the retraction would have any effect on the field.

We also asked the senior author of the paper for comment, and will update with anything we hear back.

Hat tip: Ed Yong