Hindawi reveals process for retracting more than 8,000 paper mill articles

Over the past year, amid announcements of thousands of retractions, journal closures and a major index delisting several titles, executives at the troubled publisher Hindawi have at various times mentioned a “new retraction process” for investigating and pulling papers “at scale.”  The publisher has declined to provide details – until now. 

So far in 2023, Hindawi has retracted over 8,000 articles – more than we’ve ever seen in a single year from all publishers combined. And Hindawi is not done cleaning up from paper mills’ infiltration of its special issues, according to a new report from its parent company, Wiley. 

Reckoning with Hindawi’s paper mill problem has cost Wiley, which bought the open-access publisher in 2021, an estimated $35-40 million in lost revenue in the current fiscal year, Matthew Kissner, Wiley’s interim president and CEO, said on the company’s most recent earnings call. Wiley will stop using the “Hindawi” name next year, Kissner told investors. 

The publisher has  issued a whitepaper, “Tackling publication manipulation at scale: Hindawi’s journey and lessons for academic publishing,” which explains “what happened at Hindawi” and the process the company developed to investigate and retract thousands of articles from special issues.  

According to the whitepaper, Hindawi’s research integrity staff identified “suspicious patterns” in multiple special issues around the same time as “independent researchers with an interest in research integrity, also began noticing indicators of large-scale systematic manipulation.” 

In September 2022, the publisher announced an initial batch of 500 retractions. Then, the whitepaper states: 

These initial investigations coupled with the findings of independent researchers pointed to the infiltration of Hindawi [special issues] SIs at an even greater scale than first anticipated, making it clear that thousands of manuscripts would need to be investigated. 

The publisher halted publication of special issues for a few months beginning in October 2022, and reassessed all special issue manuscripts with “a comprehensive checklist of specific hallmarks of papermill papers.” Hindawi also “carried out a thorough review and strengthening of our existing checks on editors, authors and reviewers,” according to the whitepaper. To investigate thousands of already-published papers, it developed “a new protocol aimed at detecting manipulation patterns and retracting papers rapidly and at scale.” 

Hindawi essentially created a checklist of criteria for scoring articles, and all that met a certain threshold were retracted. The whitepaper lists several “indicators of manipulation” Hindawi has been using as evidence to retract articles: 

1. Discrepancies in scope.
2. Discrepancies in the description of the research reported.
3. Discrepancies between the availability of data and the research described.
4. Inappropriate citations.
5. Incoherent, meaningless, and/or irrelevant content included in the article.
6. Compromised or manipulated peer-review.

The publisher explained: 

The decision to retract, given this evidence, is based on the rationale that the publication process has been undermined and we can no longer vouch for the integrity of the article. We have intentionally limited the specific details of what is under investigation in the retraction notice, in part because it is essential that the intelligence shared with bad actors is restricted. Moreover, to proceed expeditiously and communicate with thousands of authors, standardized wording of retraction notices was essential.

Hindawi outsourced the work of assessing individual papers to outside vendors. At least two people, trained by the publisher’s staff, evaluated each paper according to Hindawi’s criteria and filled out a questionnaire in a “bespoke software application.” In addition: 

computational tools were used to provide additional supporting evidence about fabricated content, plagiarized peer-review reports, and suspect peer review turn-around times. We also built on the valuable work done by independent research integrity sleuths, by collecting, evaluating, and categorizing comments provided on PubPeer as part of Smut Clyde’s list.

Besides retracting thousands of papers, Hindawi has used the results of its investigation to ban “several hundred” guest editors of special issues from future editorial roles and publishing articles. The publisher has also instituted “much more stringent checks” on proposals for special issues and guest editors, as well as “much greater scrutiny of peer review.” 

As well as continuing retractions, Hindawi plans to issue expressions of concern for entire special issues “to alert readers that they should take additional care interpreting all papers” when the publisher suspects more articles in a particular issue “are likely to be problematic” but has not yet investigated or found proof of issues. 

Multiple sleuths Retraction Watch asked to comment on the whitepaper complimented Hindawi for taking action and sharing information, but also questioned whether the publisher had really learned its lesson, and pointed out further work to do. 

“I think it’s a good thing they’re documenting what they’re doing and putting out information about it,” Adam Day, developer of the Papermill Alarm, said. “I hope it’s something other publishers take a lead from and feel encouraged to deal with their own problems openly as well, because this is affecting a huge number of publishers.” 

However, Day noted that he had still seen papermill articles in Hindawi journals as of this spring, after the publisher resumed special issues. “I think they’ve definitely made a big impact on the problem, but I don’t know that they’ve completely solved it yet,” he said. 

Jennifer Byrne, leader of the Publication and Research Integrity in Medical Research group at the University of Sydney in Australia, said that Hindawi’s practice of intentionally limiting the information in retraction notices should be balanced with the need for transparency. “It would be helpful for future retraction research scholars if at least some specific information about the reasons for retraction were disclosed,” she said. 

Byrne also said verifying the identities of guest editors for special issues is not enough to ensure quality, as “many guest editors could have superficially convincing credentials, ‘supported’ by publications from paper mills.” The difficulty will grow “as publications from paper mills become increasingly sophisticated and difficult to detect,” she predicted. 

Rather, the topics submitted with proposals for special issues “deserve more stringent scrutiny,” she said: 

Catch-all, duplicative special issue topics invite paper mill submissions, particularly in fields where genuine research remains very difficult, expensive and/or slow.

Cyril Labbé, who with Guillaume Cabanac and Alexander Magazinov developed the Problematic Paper Screener, praised the whitepaper’s acknowledgement of sleuths who “are working daily, most often pro-bono, to fix the work that has not been done properly by publisher.” 

Cabanac added to the whitepaper’s list of recommendations for other publishers: “fund sleuths and credit them.” 

Dorothy Bishop, who has examined paper mill activity in Hindawi special issues in depth, said she was “very pleased to see the publisher has at last grappled with the need for retractions ‘at scale.’” Banning editors and authors associated with paper mills is “a great step in the right direction, especially if they can share information about banned individuals with other publishers,” she said. 

But Bishop critiqued the framing of the document: 

The report presents Hindawi as a victim of an “academic culture of ‘publish or perish’, which has incentivized unethical behaviour”. What it omits is the influence of the commercial publisher culture of greed, which aims for massive growth in the number of published papers, with associated growth in profits. 

By her calculations, Hindawi would have brought in millions of dollars from article processing charges for now-retracted papers: “The authors of these articles don’t get their money back.”

Like Retraction Watch? You can make a tax-deductible contribution to support our work, subscribe to our free daily digest or paid weekly update, follow us on Twitter, like us on Facebook, or add us to your RSS reader. If you find a retraction that’s not in The Retraction Watch Database, you can let us know here. For comments or feedback, email us at team@retractionwatch.com.