If you notice an obvious problem with a paper in your field, it should be relatively easy to alert the journal’s readers to the issue, right? Unfortunately, for a group of nutrition researchers led by David B. Allison at the University of Alabama at Birmingham, that is not their experience. Allison and his co-author Andrew Brown talked to us about a commentary they’ve published in today’s Nature, which describes the barriers they encountered to correcting the record.
Retraction Watch: You were focusing on your field (nutrition), and after finding dozens of “substantial or invalidating errors,” you had to stop writing letters to the authors or journals, simply because you didn’t have time to keep up with it all. Do you expect the same amount of significant errors are present in papers from other fields?
David Allison and Andrew Brown: There is no way to know for sure without doing a formal assessment, but our intuition is that a priori beliefs are held passionately in the obesity and nutrition fields more so than in some other fields and we speculate that this may lead to higher rate of some kinds of problems.
RW: Sometimes you’ve been successful in getting your concerns heard – we’ve reported on multiple retractions initially flagged by your team, and cases where journals published your critical letters, including one from this week in a yoga journal. But in many cases, the journals wouldn’t publish your letters — some even asked for a fee (more than $2000, in one case) to publish a letter about a major error on one of its papers. Were you surprised by that?
DA & AB: We were struck by the temerity of the request to say the least.
RW: What was your success rate? In other words, how many papers did you try to get retracted, and how many of those were actually retracted?
We cannot precisely answer your question because: a) In some cases we explicitly recommended retraction, whereas in other cases we only implied that it might be warranted, in others we suggested retraction *OR* correction and the authors made a correction, and yet in other cases still we did not mention retraction at all. b) Several of the cases, which include several for which retraction seems likely, are still being adjudicated by the journals.
RW: You list five other barriers that you commonly encountered when trying to correct papers. For instance: Journals act too slowly, lack clear information about who to contact with concerns, are reluctant to retract faulty papers, and don’t always clearly link papers to comments associated with them, calling the results into question. Which of the barriers you identify is easiest to address? Which is hardest?
DA & AB: We think that each journal posting clear instructions on who to contact with concerns, how those concerns should be communicated, and what process will be used to address those concerns seems like an important first step that should not be unduly burdensome.
The hardest may be adjudicating situations in which retraction may be in order because this may require investigating complex scientific matters, seeking third opinions, and dealing with author concerns – all while maintaining equipoise in the rights of those raising a concern, the rights of authors, and the interests of the scientific community at large. The Committee on Publication Ethics has worked on this, and despite widespread journal adoption of COPE guidelines, journals still seem to have problems.
RW: Was there any incident you experienced with a particular journal or particular paper you were seeking to correct that stood out to you?
DA & AB: We think the sheer magnitude or absurdity of some of the errors was striking. Many of the worst involved mathematical errors. Two that stand out were:
- an unexplained baseline difference in a purportedly randomized controlled trial (RCT) that had a p-value of roughly 10^-17. To put this in perspective, if every one of the roughly 7 billion persons on the planet each did their own RCT and in each of these RCTs tested 1,000 separate variables for baseline differences, and we used a Bonferroni correction to adjust for the (1000*7 billion) tests done, a p-value of 10^-17 would still be significant.
- Weight loss and BMI loss results in a randomized controlled trial of massage therapy which, in order to be true, would have required that ordinary adult subjects would have grown over 6 cm in height during treatment.
RW: Why do you believe there are such major disincentives to correct the literature?
DA & AB: It is just unpleasant work for all concerned. No one enjoys having their errors pointed out publically. Also, literature correction is frequently conflated with misconduct, which is clearly not always the case. A paper retracted for an honest error and a paper retracted for misconduct both are labeled ‘retracted,’ so it’s easy to see how people would not want the correction of their honest errors to be associated with misconduct.
RW: What do you think about PubPeer, a forum designed to address many of the problems you raise in your Comment?
DA & AB: PubPeer and other online commenting mechanisms are great forums to discuss articles. In some cases, they have been integral to determining misconduct in publications. However, we assume that if we are unable to get results by articulating substantial errors directly to the journal, an anonymous online posting community may not have much luck, either, especially for the kinds of errors we have been identifying. The scientific community being more receptive to the idea that correction is an integral part of science may help strengthen the impact of mechanisms like PubPeer.
RW: What do you hope will result from your Comment?
DA & AB: We are hoping that authors and editors realize how common such invalidating errors are, that authors will begin more careful processes of checking their own work especially around math and statistics, and that journal editors may be more sensitive to the need to evaluate and act on post-publication criticisms of papers.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy.
Howdy – Your link to the Nature commentary is broken. So is the link from Nature’s online table of contents (http://www.nature.com/nature/current_issue.html). I finally found it at http://www.nature.com/news/reproducibility-a-tragedy-of-errors-1.19264
Ken
Fixed, thanks!
In relation to PubPeer’s possible influence, I would like to compare this to the experiences in the computer security community. Security researchers would spend significant amounts of time communicating privately with companies and see little or no response. Then they started posting their findings publicly and suddenly the problems were fixed quickly. Even anonymously posted problems that were publicized by others were dealt with in a timely fashion. Now obscure research results aren’t the same thing as bugs in widely used web browsers or operating systems, but I think this example still supports the idea that public (even anonymous) airing of concerns about research results could be more effective than private communications.
The authors say “However, we assume that if we are unable to get results by articulating substantial errors directly to the journal, an anonymous online posting community may not have much luck, either…”
Posting concerns publicly has two (non-exclusive) advantages:
– the publicity puts welcome pressure on the parties dragging their feet
– the readers get to know about the problems immediately
Hats off to Andrew Brown and David Allison, though as their experience, and that of many others teaches us, journals seem to be very reluctant to retract. Nice bit of evidence from another post on RW
http://retractionwatch.com/2016/02/03/2001-fujii-papers-retracted-finally-what-took-so-long/
where even the presumably keen sighted British Journal of Ophthalmology have just retracted a 17 year old paper from a well publicised multiple offender, Yoshitaka Fujii.
I expect most of the open-access journals that require article processing charges (APCs) also charge to publish letters to the editor pointing out mistakes in their published articles. A colleague of mine and I ran into this, which led to checking policies at other reputable, OA journals. Some required the full fee, others a reduced rate, but of those I checked, none would formally publish a letter pointing out needed corrections without a fee. I generally could not tell this from their online guidelines, but had to write the editor to find out. Presumably, publishers don’t want to start down a slippery slope of fee exceptions. However, requiring a substantial fee to publish an open, editor-moderated correction letter has its own ethical issues. At the least, it certainly discourages letters to the editor pointing out errors, omissions, or alternate explanations that warrant being linked to the original article, even if the don’t rise to the retraction level.
A colleague and I have a commentary in review on our experiences and could pass it along if/when it is published if there’s interest.
I do sincerely hope that the irony of this being published in one of the most recalcitrant journals on the planet, is not lost on the readers of the article.
There is another issue here that we simply must discuss as a community. When journals have publications retracted, or when institutions are found to have hired a fraudulent scientist, many people act as though this reflects badly on the journal or institution. This creates huge incentives for covering up or downplaying misconduct.
This same “cone of silence” creates other problems. The one that comes to the front of my mind is how victims of rape are sometimes treated by administrators on college campuses.
There is no institution on earth that is totally devoid of bad apples, and we need to create incentives to do the right thing. The only “bad PR” should be what happens when you do the wrong thing. We should celebrate the ethics of journals who retract fraudulent work and institutions who fire fraudsters.