Don’t like a paper, but don’t want to retract it? Just issue an “editorial statement”

Last April, the American Journal of Epidemiology and the American Journal of Public Health published a rare joint editorial statement. It concerned a pair of papers on the topic of mortality and obesity. Several complaints had prompted the journals to investigate. Their assessment: These papers contained inaccurate results.

The statement was not a retraction—it was a compromise the editors came up with that would set the academic record straight, while not tainting the authors’ publication record, given that they had (in the editors’ opinion) made honest mistakes. It was an unusual solution to a not-uncommon problem (criticisms of a paper), in which the editors tried to balance their duty to the scientific record against its potential impact on the authors. And it left few people happy — including researchers in the field, who are left unsure about the validity of the results.

Roland Sturm, an economist at Pardee RAND Graduate School who was not a co-author on the papers, told Retraction Watch:

…I am getting new journal articles to review, and they still cite these articles. and I went back to check and there’s nothing flagging these articles, and it’s not okay.

Back and Forth

Published five years ago, the papers used a novel method to conclude obesity is much more fatal than earlier studies suggested. The papers received a fair amount of coverage at the time, in publications like Scientific American, USA Today, CNN, and others. Ryan Masters, then a postdoc at Columbia University, and now an assistant professor at the University of Colorado-Boulder, was lead author on the papers.

The first paper appeared in the February 2013 edition of the American Journal of Epidemiology (AJE), and attracted some attention from the scientific community, notably from Zhiqiang Wang, an associate professor in epidemiology at the University of Queensland in Australia. He penned a letter to AJE in 2014, pointing out what he saw as fundamental errors in the way the study had correlated its data. Wang told us:

When I first read the paper, I felt that their findings were important if confirmed. So, I recalculated age-specific hazard ratios according to the coefficients provided in their articles. However, my results contradicted their findings and conclusions.

Masters et al responded to Wang’s critiques, and to others. Masters told us:

We knew it was a contentious position, pushing a new methodology. We sort of welcomed the discussion that followed.

Later that year, in October, Masters and his collaborators’ second paper appeared in the American Journal of Public Health (AJPH). This one used the methodology they’d established in the AJE study to suggest that being overweight or obese was particularly harmful to older adults; for people between 40 and 85, obesity was four times as fatal as suggested in other studies. Wang saw that paper, and wrote a second letter, this time to the editors of AJPH, pointing out the similar  problems to those he perceived in the earlier AJE paper.

The AJPH paper also caught the attention of Katherine Flegal, who led several studies while at CDC’s National Center for Health Statistics that had come to opposite conclusions. When she took a long, hard look at the AJPH paper, she noticed an error in a formula calculating attributable risk—a wholly different issue than the one Wang was writing about. She told us:

I wrote to [Masters], “I don’t understand the quantities in this formula.” He gave me an answer that wasn’t easy to understand, so I was still puzzled. I wrote back, and he didn’t answer. I wrote back again, and occasionally again, and he never answered. So I got frustrated one morning and wrote to the journal.

That morning was January 13, 2015. The email went to Mary Northridge, who was outgoing editor in chief of AJPH. She notified Masters and asked for his response, and forwarded the exchange along to Alfredo Morabia, who was in the process of taking over as editor in chief of the journal. Both are faculty at Columbia University’s Mailman School of Public Health.

Masters and his co-authors conceded to Northridge that the disputed table from the AJPH paper indeed contained a flawed formula. At the journal’s behest, they began crafting an erratum to address the problem—by substituting a different formula. They did not think the error sunk their paper. So, they asked Morabia if they could include an “On The Other Hand”—a feature of the journal that allows counterarguments to published research—explaining their correction in context. Morabia agreed.

However, soon after granting that request, Morabia began looking into a third letter from Wang, the second submitted to AJPH, to which Masters and his co-authors had declined to respond. (Said Masters: “We’d addressed his points twice already [in AJE], and at this point felt like we were beating a dead horse.”) However, Morabia found Wang’s letter compelling, and was put off by the fact that Masters refused to respond. So, Morabia recruited James Hanley, a biostatistician from McGill University, to review Masters’ original research.

In late 2015, Masters and his colleagues submitted to AJPH the Erratum and the On The Other Hand—which only responded to the flawed formula Flegal had pointed out. In January 2016, Morabia informed them the journal wouldn’t publish the “On the Other Hand.” He told them Hanley had found their model flawed, and results untrustworthy.

“Significantly altered”

Morabia still published the erratum—but deleted two final sentences that the authors felt were crucial. These read:

We maintain that the PAFs from equation (1) that were presented in Masters et al. 2013 are preferred because these newly estimated PAFs from equation (2) are biased. To see how PAFs estimated from equation (2) are biased please see our “On the Other Hand” accompanying this erratum.

Masters and his co-authors were incensed when they saw the erratum in print. Morabia hadn’t informed them of the edit beforehand.

Masters told us:

Dr. Morabia’s editorial decision made it look like we admitted fault.

In response, he and his co-authors filed an ethical complaint about Morabia to the American Public Health Association, AJPH’s parent organization.

The APHA sided with Morabia, telling Masters and his co-authors that they give their editors a large degree of freedom in the content that goes into the journal. Regarding his decision to edit erratum, Morabia told Retraction Watch:

The erratum is an issue of maintaining the integrity of the journal. It’s an editorial piece, not an opinion piece from the authors. I have the right to decide what should be in the erratum.

To Morabia, the erratum addressed an irrelevant issue—the single formula—given that Hanley’s analysis was pointing out fundamental flaws that invalidated the paper’s whole model. Morabia did not want force a retraction. Instead, the editor and his co-evaluators cooked up a less drastic alternative: The joint editorial. Months earlier, he had notified his counterpart at AJE, Moyses Szklo, an epidemiologist at Johns Hopkins Bloomberg School of Public Health, about his growing concerns with the 2013 AJPH paper. Szklo directed his own review of the companion piece, the 2013 AJE paper. The two editors came to similar conclusions—both papers were fundamentally flawed — specifically due to the methodological issues Wang pointed out in his letter, not the formula Flegal flagged. According to Morabia, it was Szklo who suggested the joint editorial statement. Morabia told us:

This was not an issue of misconduct…So we decided that the priority was not blaming the author, but transparency.

Editorial Superposition

That joint editorial went live in April 2017, in the same issue of AJPH as Hanley’s 1200 word critique of Masters et al. Morabia gave Masters and his co-authors 600 words to rebut Hanley (the editor says this disparity in word counts is the journal’s typical format for exchanges like this), most of which they used to instead exhort readers to read a 5,000 word rebuttal published on Masters’ personal website. Word count isn’t the only way the authors felt cheated. Masters’ co-author, Bruce Link, an epidemiologist at Columbia University’s Mailman School of Public Health, contends Hanley’s paper was not formally peer reviewed, and is therefore an unequal counterweight to he and his co-authors’ peer reviewed work.

Wang, Flegal, and Sturm noted their dissatisfaction with how the journals handled this situation, most saying they’d hoped for a clearer outcome. “Personally, I think it would have been better if it had been retracted,” says Flegal. She says she’s never witnessed a journal addressing a problematic paper in such a way, and expressed doubts that it would serve its intended purpose of warning readers about the findings.

The AJE paper, “Obesity and US mortality risk over the adult life course,” has been cited 54 times — 49 times since 2014, according to Clarivate Analytics’ Web of Science. The AJPH paper “The impact of obesity on US mortality levels: the importance of age and cohort factors in population estimates” has been cited 115 times since it was published in 2013; all but one citation occurred since 2014, when Wang submitted his letters about the work. Both are considered to be Highly Cited papers, meaning  they ranked in the top 1 percent of all papers in their field for the years they were published.

As of press time, the versions hosted on AJE, AJPH, and PubMed contain neither links nor statements indicating the editors’ concerns. Morabia told Retraction Watch that this was an oversight, and he would look into getting the link added to the AJPH paper’s page. AJE editor Szklo declined to speak with us.

To the authors of those papers, this is evidence of yet more bad faith on behalf of the journals. “I think the record has been unfair,” said Link. “But everyone should have access to it.”