Political scientist asks for correction, gets flip-flop

You’d think that if an author asked a journal to correct a modest mistake, the journal would oblige. After all, many researchers have to be dragged kicking and screaming to correct the record.

But for Ryan Enos, a political scientist at Harvard, self-correction of a paper he had published in the Proceedings of the National Academy of Sciences (PNAS) took far more steps than one might have hoped.

It’s probably easiest to let Enos tell the story, which he did earlier this week in a series of tweets:

https://twitter.com/RyanDEnos/status/1072594306790244353

https://twitter.com/RyanDEnos/status/1072595028722286592

https://twitter.com/RyanDEnos/status/1072595376853671941

Here’s the correction he sent PNAS on December 6:

On the survey administered to subjects, there were three questions about immigration policy. These are correctly listed on page 3701 as:

i) “Do you think the number of immigrants from Mexico who are permitted to come to the United State to live should be increased, left the same, or decreased?”

ii) “Would you favor allowing persons that have immigrated to the United States illegally to remain in the country if they are employed and have no criminal history?”

iii) “Some people favor a state law declaring English as the Official Language. Some other people oppose such a law. Would you favor such a law?”

In Table 1 on page 3701 and Figure 2 on page 3702, question ii is mistakenly recorded as “Children of undocumented be allowed to stay?” The actual question asked of subjects is correctly listed as the question in the text on page 3701.

Here’s how PNAS responded a day later:

Thank you for letting us know about this. For errors of scientific content, PNAS publishes Corrections only if the error would substantially impact the results or conclusions, or if an informed [Ed: reader] would be hindered from understanding the text or reproducing the results. While the error is unfortunate and a Correction would address the matter, the fact that the correct question is included in the text, just not in Table 1 or Fig. 2, means a reader should be able to infer the correct meaning, and would not be significantly misled by the error. As such, the error does not require publication of a Correction. Should any readers contact us about the error, we will direct them to contact you via the email address on the first page.

That seemed a bit odd to us, so we asked PNAS why they hadn’t just corrected the paper. Our email seems to have prompted a reconsideration:

We consider correction requests on a case-by-case basis. In this case, the author notified us of the error in the figure and table but also indicated that the correct information was included in the text and that the error did not substantively or quantitatively alter the findings of the article. Given that the error in the figure and table could be potentially misleading to readers, we will reach out to the author to publish a correction. Thank you for bringing this matter to our attention.

We asked Enos what he thought of the fact that PNAS is now correcting the paper:

A general point, which some have raised in response to my tweet, is that, perhaps journals should have some sort of “version control” for cases like this where there is an error that should be corrected but it perhaps isn’t substantial enough to change results and need a correction or retraction.

He pointed to an example involving a former colleague:

As I understand it, the situation was that the journal typesetters actually retyped his LaTex table and did so incorrectly.  As such, after publishing the article, they discovered his replication material didn’t match the published article, so they retracted the article, even though his replication material matched the paper that was accepted to the journal (just not the one they typeset).  Seems like the right thing to do would have just been to update the article.

Obviously, there are limits to this, of course, you don’t want to incentivize researchers to be sloppy, knowing that they can always correct mistakes, but there might be a middle ground.

We asked Enos what else he had to say about the incident:

It is ultimately a fairly inconsequential (in that the inferences from the paper don’t really change), but rather telling mistake.

I suppose what happened is that the first time I made the figures and tables in R, in my head I had a different question than the one that was actually asked on the survey that was administered to subjects.  I made those figures, put them in the paper, and never noticed the mistake. The paper went through reviews and editing and nobody noticed the discrepancy. I have given dozens of talks including the paper, many to people who have read the paper and nobody ever mentioned it, scholars have even tried to replicate the experiment and never asked me about it.  I even reproduced the figure in my book, so I the error reproduced itself there. It wasn’t until Jordi Honey-Roses at the University of British Columbia emailed me with questions about the paper that somebody finally raised the inconsistency.

I think this is telling because 1) it shows how easily errors can reproduce themselves – I never noticed this rather glaring inconsistency, so think of how easy it is to miss, say, a more obscure error in code. That is one reason it is important to make your code public, so that others can see the errors that we can’t see because we often have blinders when it comes to our own work.   2) It shows that this only works if people speak up about errors they find. Surely, somebody had noticed this particular error before. It’s a fairly well-cited paper and I know it is assigned pretty widely in courses. Maybe people thought they error was inconsequential so they didn’t want to bother raising it. But I’m rather embarrassed it happened and I wish one of the people who saw it would have told me earlier.  Science is only worthwhile if it is accurate, so better to get it accurate earlier rather than later.

Indeed.

Like Retraction Watch? You can make a tax-deductible contribution to support our growth, follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up for an email every time there’s a new post (look for the “follow” button at the lower right part of your screen), or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].