In our line of work, we see it all — mega-corrections that don’t quite rise to the level of retraction, letters to the editor that point out seemingly fatal flaws in papers that remain untouched, and studies retracted for what seem like minor reasons. It can make you wonder what makes a paper worthy of a retraction. A recent case in an obesity journal may not provide a definitive answer, but it gives us a lot to chew on.
Here’s the story: In September 2013, Rosely Sichieri and a colleague from the State University of Rio de Janeiro submitted an article to Obesity Facts, “Unbalanced Baseline in School-Based Interventions to Prevent Obesity: Adjustment Can Lead to Bias?” The article examined statistical issues in randomized controlled trials of school-based weight loss programs. Peer reviewers said the paper needed major revisions before it could be accepted; the authors revised the paper enough in a second draft, submitted in November 2013, that the original reviewers accepted it. The paper was published in June 2014.
Then, in September 2014, a group of authors including David Allison of the University of Alabama, Birmingham, and colleagues from Clemson, Thomas Jefferson, and the University of Minnesota, wrote a critical letter that was published in the journal in April. The letter, according to a just-published editorial:
expressed fundamental and severe criticism with regard to the above mentioned article that culminated in the conclusion that the article should be retracted.
More specifically, the letter argued that by criticizing some of the statistical tools used in these types of studies, the authors dissuade scientists from employing “legitimate power-enhancing analytic methods.” Here’s more from the letter itself:
The fundamental conclusion as stated in its title and elsewhere in the article is incorrect. For example, the statement ‘Although adjusting for the baseline values of parameters (sic, variables – Li et al.) that are highly influenced by baseline values is a standard procedure, this approach can bias the results …’ is simply untrue. Such erroneous conclusions could lead researchers to avoid legitimate power-enhancing analytic methods, and should be retracted.
Sichieri and her colleague responded to that letter, calling the criticisms “unfair” and saying that:
Our study cannot be retracted exactly because it indicates this possibility of bias. We are not arguing that randomization procedure with unbalanced baseline is due to chance, therefore there is no selection bias. However, in this situation, usually in school-based interventions to prevent obesity, adjustment may have biased the results.
In an editorial in the current issue, Hans Hauner, the editor of the journal, describes the critical letter and author’s response, including “their reasons for refusing to retract their article.” Because the letter’s criticism was “so severe,” an independent statistician was asked to review the material; that statistician agreed that the original authors have “not adequately addressed” the letter’s criticisms. Hauner concludes:
By publishing the Letter to the Editor and the Authors’ Response together with this Editorial in the current issue of Obesity Facts, I would like to open the discussion that arose between the authors of the original article, the authors of the Letter to the Editor and the Editor-in-Chief to our readership and recommend reading and forming your own opinion.
Stimulated by this debate, I would like to point out that it is each author’s responsibility to make sure that statistical procedures are correctly used and valid for the study submitted. Even though it would be desirable if an additional assessment of statistical methods could be established within the review process, it cannot be expected that all of our reviewers, in addition to their expertise in various aspects of obesity research, are designated experts for advanced statistical procedures.
Finally I would like to emphasize the importance of critically scrutinizing scientific papers even after their publication. Therefore, comments on articles are highly welcomed at all times to present and discuss different viewpoints concerning a controversial topic.
We asked Allison whether he was satisfied with the journal’s response:
I believe that the journal handled the matter respectfully, transparently, and in a manner that promoted dialogue – and that is all to the good. That said, I also believe that journal editors have the authority to retract papers in their journals. In this case, the editor had information from an independent third party expert that the original paper provided readers with erroneous conclusions and merited retraction. Therefore, I believe that the journal should have retracted the original paper.
We’d love to hear what Retraction Watch readers think.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.
As a statistician, if I had come across the paper by Sichieri and Cunha, I would have dismissed it as a poorly written opinion piece, certainly not any serious piece concerning statistical methodology or analysis. I would also have concerns about the level of quality of peer review for the journal involved.
If an article’s thrust concerns advanced statistical procedures, then a reviewer with advanced statistical expertise should be sought out. If no such reviewer can be found, the editor should refer the authors to publish elsewhere, rather than accepting an article that purports to review complex statistical issues with no solid review before publication.
The editor was able to find a statistician to review this matter after the fact: “As the criticism expressed in the Letter to the Editor is so severe, an independent statistician was asked to review the original article and the subsequent correspondence. The statistician agreed with the criticism raised by Li et al. [2] and concluded that the critical comments were not adequately addressed in the Authors’ Response [3]”. Why wasn’t this done before publishing the article? This whole episode could have been avoided.
This may be a reflection of the current commercialization of journals, wherein commercial journals now publish articles to fulfil financial goals; compared to the old school society journals, whose goals used to include the dissemination of useful scientific material.
Reviewing the original PLOS ONE article, I find it also to be poorly written and reviewed. I can find no description of their statistical models, neither in the PLOS ONE article, nor in any of the supplemental materials; only vague references to SAS statistical computer programs. Full disclosure of the models fitted, the statistical algorithms used to fit the model to the data, and the statistical tests yielding the quoted p-values, are essential to properly evaluating the results, including the issue of adjustment for baseline BMI (BMI, being a ratio variable, having statistical problems itself, but that’s another story).
The table of results in the PLOS ONE article (Table 2) is too vague to allow any serious evaluation of their methodology or results. A proper statistical assessment of the intervention effect would test both the interaction effect and the intervention effect in an omnibus test, but the authors only discuss tests of the interaction effect. The interaction term p-value of 0.02 obtained when adjusting for baseline measures is not that small, and there is no discussion of what practical significance the model parameter estimate of 0.003 (unadjusted model) or 0.006 (adjusted model) represents in terms of e.g. change in BMI. The problem of confusing statistical significance with biological relevance ensues. At least the authors did not focus on that p = 0.02 finding, though for inappropriate reasons.
It is unfortunate that modern publishing, with its space limitations and financial directives, yields so many articles of poor quality and limited, if any, scientific merit.
If the Obesity Facts article was retracted, should the PLOS ONE article also be retracted? In that case, how many other poorly written articles should be retracted? At this point I think we are stuck with heaps of poor journals publishing piles of unsound articles. It’s going to take a long time to sort this mess out, and I think documenting which journals and articles show little scientific merit will be more useful than sweeping them under the rug. The whole system needs reworking, and hiding a few poorly written articles isn’t going to fix it.
At the very least a link to the criticism and discussion thereof should be placed prominently on the article page. Having an open and transparent conversation is only useful if people are aware it is going on.