A paper in Contraception that purported to show serious flaws in an earlier study of abortion laws and maternal health has been retracted, after the authors of the original study found what were apparently significant flaws in the study doing the debunking.
That’s the short version of this story. The longer version involves years of back-and-forth, accusations of conflict of interest and poor research practice, and lawyers for at least two parties. Be warned: We have an unusual amount of information to quote from here that’s worth following.
As the editor of Contraception, Carolyn Westhoff, put it:
I got to make everybody angry.
‘An accusatory or ad hominem tone’
The story begins four years ago, when BMJ Open published “Abortion legislation, maternal healthcare, fertility, female literacy, sanitation, violence against women and maternal deaths: a natural experiment in 32 Mexican states,” by Elard Koch and his colleagues. That paper was accompanied by a press release from the MELISA Institute titled “Study Finds Better Maternal Health With Less Permissive Abortion Laws.”
Koch, founder and director of the MELISA Institute, which says it studies “determinants of maternal, embryonic and fetal health from an epidemiological and biological perspective,” told Retraction Watch that
the purpose of our study published in BMJ Open was not to argue for or against abortion laws, but to examine its association with the [maternal mortality rate] MMR in Mexican states after controlling for a number of factors known to impact maternal health at the population level. Our findings support the null hypothesis: after controlling for different combinations of these factors (we used a panel of 24 regression models), there was no evidence supporting an independent association (whatever positive or negative) between Mexican abortion legislation and MMR was found in our study.
The MELISA Institute does not seem to be well-known among researchers who study abortion policy; three such researchers contacted by Retraction Watch had not heard of the organization before reading about these two papers.
The saga began in earnest in August 2016, when Contraception published “Maintaining rigor in research: flaws in a recent study and a reanalysis of the relationship between state abortion laws and maternal mortality in Mexico,” by Blair Darney, of Oregon Health & Science University, and colleagues. The authors conclude:
We support a recent call to improve abortion data and research by adhering to three criteria: transparency, acknowledging the limitations of data and contextualizing results. Koch and colleagues fail at all three and do not help us understand the relationship between decriminalization of or access to safe abortion and women’s health.
That paper, Koch said,
aimed to refute and apparently discredit our BMJ Open paper based on the data from Mexico using an accusatory or ad hominem tone. We infer the specific purpose to discredit our research from the original research proposal funded by a $250,000 grant to Dr Darney by the Society of Family Planning. The SFP proposal states that they ‘have failed to respond to anti-abortion “junk science,” which influences policy in the region”. Sincerely, I don’t know how our study is ‘influencing’ policies in the region and of course I don’t consider our work as “junk science”.
When Koch and his colleagues reviewed the paper, they found an error.
According to the result presented in the Table 2 of the paper, there was a decline in maternal mortality associated with 31 states outside Mexico City, with restricted access to abortion (beta = -22.49) but the authors interpreted the result in opposite way, that is, associated to a decline of MMR in Mexico City with wide access to abortion.
When they tried to replicate the findings, Koch confirmed their original reading of the paper, and they submitted a manuscript describing their replication to Contraception in May 2018.
Our replication study confirmed that the negative regression coefficient (beta = -22.49) presented in the Contraception paper was basically correct, and we ruled out any typographical error for this result. In addition, we detected other serious methodological flaws and omissions in the paper. Overall, we found evidence supporting a potential case of research misconduct. The ad hominem accusations of a lack of transparency and false conclusions they stated for our study were unsupported and untrue.
An insufficient correction
On August 10, Westhoff told Koch by email that there was in fact an error in the Darney et al paper.
Dr. Darney has confirmed this, and my current plan is for her to submit an erratum to the journal. This erratum will acknowledge that this error was identified by a careful reader. I will let you know when we receive an erratum. If I find this to be satisfactory, then we will publish the erratum, but I do not see the need to publish an additional full-length paper on the topic.
Koch wrote back on August 13 to say that an erratum would not be enough:
The number and magnitude of the analytical errors detected in our replication study includes a serious misinterpretation of a pivotal result. This misinterpretation essentially invalidates the assumptions and conclusions from Darney et al‘s paper. We feel that this fundamental error cannot be corrected through an erratum, because unfortunately, it is neither minor nor simple.
In fact, it completely changes the conclusion of the paper. In addition, we remain concerned about the tone of Darney et al’s paper. The authors questioned the integrity of our BMJ Open publication, and their accusations are unsupported in light of their own study’s methodological flaws. An erratum cannot address this, or the damage to our research reputation, which has already occurred, since this article appears along with our other work during Medline searches. In our opinion the only acceptable and proportional solution in this case is that the authors proceed with a RETRACTION of their article. If they do not retract the paper then the editors of Contraception should do so. In addition, we believe this retraction should be accompanied at least with a reply or editorial comment pointing out the large issues leading to retraction.
It is of some concern that the manuscript was led and submitted by a current member of the editorial board of Contraception, and the research itself was supported for a grant from the same institution funding the journal. However, it is an important and valuable first step to know that Dr Darney recognizes an error in their article (at this moment we don’t know the opinion of her colleagues). We need emphasize that there are a number of elements, which might potentially lead to a request for an investigation by the Office of Research Integrity and perhaps by COPE as well. In this context, we would sincerely prefer to work from the assumption that honest mistakes were made in writing this article, that these errors were not detected during an independent and fair peer review and publication process by Contraception, and that there were no other motives. An immediate retraction would be a clear signal of good faith and acknowledgement by the authors in the right direction, avoiding the burden of additional proofs.
On August 30, Andrea Bocelli, a publisher at Elsevier, wrote back to Koch:
Contraception is wholly owned by Elsevier Inc. Elsevier and our Editors are members of the Committee on Publication Ethics (COPE). Elsevier is not involved in the editorial decisions of the Journal and neither are our affiliated societies.
Dr. Darney is currently on the Board of Contraception, but she was not at the time her paper was accepted. Even so, it is important to note that work submitted to the Journal by Editors or Board members is assessed using the same criteria as that applied to all Contraception submissions.
I understand you feel an erratum is not sufficient. Dr. Darney did acknowledge an error in one of her tables and supplied a correction. She feels this correction does not change the overall conclusion of her paper. In order to review your allegations we are seeking a neutral review by an independent third party. Dr. Westhoff and I thank you for your patience while this investigation is being conducted.
Koch and his co-authors wrote back on September 8, detailing their concerns. On October 31, Westhoff emailed Koch to say that the journal had reversed its decision:
We have evaluated Dr. Darney’s 2017 paper, and decided that the paper requires retraction.
Sometime between then and December 4 — it is unclear when, as Elsevier did not add a date to the retraction notice — the paper was retracted:
This article has been retracted at the request of the Editor-in-Chief and Authors.
The authors recently discovered an error that affected the results in their article on the relationship between state-level maternal mortality in Mexico and state-level abortion legislation. In Table 2 the beta-coefficient for abortion legislation was calculated as -22.49 and erroneously interpreted as +22.49. This error affects several of the paper’s conclusions, and thus the editor and authors have jointly made the decision to retract the paper.
The authors would like to express their sincere regret at the errors in their initial report.
That retraction notice didn’t satisfy Koch and his colleagues. By this time, they had hired Paul Thaler, of Cohen Seglias, a law firm in Washington, DC. Thaler is perhaps best known for representing researchers who are accused of scientific misconduct. On December 4, Thaler wrote a letter to Contraception suggesting a different retraction notice:
This article has been retracted: please see Elsevier Policy on Article Withdrawal (https://www.elsevier.com/about/our-business/policies/article-withdrawal).
This article has been retracted at the request of the Editor-in-Chief and Authors.
The article purported to replicate, reanalyze and provide a critical review of a previous study on the relationship between state-level maternal mortality ratio (MMR per 100,000 live births) in Mexico and state-level abortion legislation [Koch E, Chireau M, Pliego F, Stanford J, Haddad S, Calhoun B, Aracena P, Bravo M, Gatica S, Thorp J. Abortion legislation, maternal healthcare, fertility, female literacy, sanitation, violence against women and maternal deaths: a natural experiment in 32 Mexican states. BMJ Open 2015;5(2):e006013]. The authors of the BMJ Open article conducted a thorough replication study of the now retracted paper, and they submitted their findings to the editor-in-chief of Contraception. An independent and neutral statistical review commissioned by the editors corroborated several methodological flaws, including a serious misinterpretation in the beta coefficient for abortion legislation in Table 2 of the now retracted article. This coefficient was calculated as -22.49 decline in the MMR associated to the 31 states with restricted access to abortion, but the authors erroneously interpreted this result as associated to Mexico City with wide access to abortion. This and other major errors affect several of the paper’s conclusions, and thus the editor and authors have jointly made the decision to retract the paper.
The authors would like to express their sincere regret at the errors in their initial report. The retraction of the article removes the basis the authors relied on for criticizing the BMJ Open study.
Elsevier rejected that suggestion. In a December 21 email, associate general counsel Jessica Alexander wrote:
In terms of the timeliness of the journal’s investigation process, as the wording of your proposed retraction notice acknowledges, the journal conducted an independent and neutral statistical review. This required the involvement of an independent external statistician as well as editorial review, which inevitably takes some time. The Editor took this issue seriously and the goal was to undertake a thorough review. This resulted in the decision to retract the article.
As set out in the Retraction Guidelines from the Committee on Publication Ethics, the purpose of retractions is not to punish authors but rather to correct the literature. The notice should mention the reasons and basis for retraction and who is retracting the article. This notice includes a description of the error that is the basis of the retraction and also includes a clear apology from the authors. It conforms to Elsevier’s standards and it alerts readers to the error. I am afraid that we do not therefore agree that the retraction notice requires amendment.
The editor of the journal has already declined your clients’ request to publish a response to the Darney article. Elsevier has a policy of editorial independence and we support this decision of the editor.
Koch tells Retraction Watch that he and his colleagues are considering filing complaints with the ORI and COPE.
Asked for comment, Darney referred us to the journal, and to OHSU’s communications office. An OHSU spokesperson said that Darney and the university declined comment.
A first for journal editor
While Contraception has had retractions before, Westhoff, who has been editor for five years, said this was the first she had handled. (There was a temporary removal in 2016, which Elsevier does not classify as a retraction.)
I struggled, never having written one before, with what should go into a retraction notice. I think the goal is to correct the record. And I think it did that.
If the authors are still angry, that’s a different level. I don’t think it’s my job to assuage their anger, which may not be possible.
Westhoff suspects that her decision not to publish the Koch group’s rebuttal to the now-retracted paper — a rebuttal that seemed to be moot, once the paper was retracted — was “infuriating.” But the claims of a conflict of interest don’t hold water, she said. Yes, she was once president of the society that funded Darney’s analysis, but the board of directors doesn’t have anything to do with such funding decisions, Westhoff said. And Darney only joined the editorial board of the journal after the now-retracted paper was published.
They’re trying to make some connections there that are at best really tenuous.
At the end of the day, said Westhoff,
The author made an error, and we made an error, in that the peer reviewers missed it. It won’t be the last time.
In retrospect, there was some inflammatory language in the paper that I might have suggested changing. There are a lot of papers and I do miss things. The peer reviewers are still unaware of all of this. I think there were errors here but they were honest ones.
It seems that hardly anyone noticed the papers, which Westhoff said she found a relief.
We did look at citations and social media and we saw essentially no evidence that either of the papers got any traction. I think the only person who cited [the Darney et al paper] was the author herself.
Like Retraction Watch? You can make a tax-deductible contribution to support our growth, follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up for an email every time there’s a new post (look for the “follow” button at the lower right part of your screen), or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at firstname.lastname@example.org.