If you notice an obvious problem with a paper in your field, it should be relatively easy to alert the journal’s readers to the issue, right? Unfortunately, for a group of nutrition researchers led by David B. Allison at the University of Alabama at Birmingham, that is not their experience. Allison and his co-author Andrew Brown talked to us about a commentary they’ve published in today’s Nature, which describes the barriers they encountered to correcting the record.
Retraction Watch: You were focusing on your field (nutrition), and after finding dozens of “substantial or invalidating errors,” you had to stop writing letters to the authors or journals, simply because you didn’t have time to keep up with it all. Do you expect the same amount of significant errors are present in papers from other fields?
David Allison and Andrew Brown: There is no way to know for sure without doing a formal assessment, but our intuition is that a priori beliefs are held passionately in the obesity and nutrition fields more so than in some other fields and we speculate that this may lead to higher rate of some kinds of problems.
RW: Sometimes you’ve been successful in getting your concerns heard – we’ve reported on multiple retractions initially flagged by your team, and cases where journals published your critical letters, including one from this week in a yoga journal. But in many cases, the journals wouldn’t publish your letters — some even asked for a fee (more than $2000, in one case) to publish a letter about a major error on one of its papers. Were you surprised by that?
DA & AB: We were struck by the temerity of the request to say the least.
RW: What was your success rate? In other words, how many papers did you try to get retracted, and how many of those were actually retracted?
We cannot precisely answer your question because: a) In some cases we explicitly recommended retraction, whereas in other cases we only implied that it might be warranted, in others we suggested retraction *OR* correction and the authors made a correction, and yet in other cases still we did not mention retraction at all. b) Several of the cases, which include several for which retraction seems likely, are still being adjudicated by the journals.
RW: You list five other barriers that you commonly encountered when trying to correct papers. For instance: Journals act too slowly, lack clear information about who to contact with concerns, are reluctant to retract faulty papers, and don’t always clearly link papers to comments associated with them, calling the results into question. Which of the barriers you identify is easiest to address? Which is hardest?
DA & AB: We think that each journal posting clear instructions on who to contact with concerns, how those concerns should be communicated, and what process will be used to address those concerns seems like an important first step that should not be unduly burdensome.
The hardest may be adjudicating situations in which retraction may be in order because this may require investigating complex scientific matters, seeking third opinions, and dealing with author concerns – all while maintaining equipoise in the rights of those raising a concern, the rights of authors, and the interests of the scientific community at large. The Committee on Publication Ethics has worked on this, and despite widespread journal adoption of COPE guidelines, journals still seem to have problems.
RW: Was there any incident you experienced with a particular journal or particular paper you were seeking to correct that stood out to you?
DA & AB: We think the sheer magnitude or absurdity of some of the errors was striking. Many of the worst involved mathematical errors. Two that stand out were:
- an unexplained baseline difference in a purportedly randomized controlled trial (RCT) that had a p-value of roughly 10^-17. To put this in perspective, if every one of the roughly 7 billion persons on the planet each did their own RCT and in each of these RCTs tested 1,000 separate variables for baseline differences, and we used a Bonferroni correction to adjust for the (1000*7 billion) tests done, a p-value of 10^-17 would still be significant.
- Weight loss and BMI loss results in a randomized controlled trial of massage therapy which, in order to be true, would have required that ordinary adult subjects would have grown over 6 cm in height during treatment.
RW: Why do you believe there are such major disincentives to correct the literature?
DA & AB: It is just unpleasant work for all concerned. No one enjoys having their errors pointed out publically. Also, literature correction is frequently conflated with misconduct, which is clearly not always the case. A paper retracted for an honest error and a paper retracted for misconduct both are labeled ‘retracted,’ so it’s easy to see how people would not want the correction of their honest errors to be associated with misconduct.
RW: What do you think about PubPeer, a forum designed to address many of the problems you raise in your Comment?
DA & AB: PubPeer and other online commenting mechanisms are great forums to discuss articles. In some cases, they have been integral to determining misconduct in publications. However, we assume that if we are unable to get results by articulating substantial errors directly to the journal, an anonymous online posting community may not have much luck, either, especially for the kinds of errors we have been identifying. The scientific community being more receptive to the idea that correction is an integral part of science may help strengthen the impact of mechanisms like PubPeer.
RW: What do you hope will result from your Comment?
DA & AB: We are hoping that authors and editors realize how common such invalidating errors are, that authors will begin more careful processes of checking their own work especially around math and statistics, and that journal editors may be more sensitive to the need to evaluate and act on post-publication criticisms of papers.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy.