Revealed: The inner workings of a paper mill

In 2019, Retraction Watch ran an exclusive story of a Russian paper mill operating under the business name “International Publisher LLC”.  Since then, Retraction Watch and  other scientific news and blogging sites have continued to report on the activities of research paper mills, including International Publisher  and its primary website, 123mi.ru.  These mills provide an array of fraudulent services to researchers and academics seeking to publish articles in peer-reviewed journals.  The services they provide include ghostwriting, brokering authorship positions on papers accepted for publication, and falsifying data.

Our project  augments this stream of reports about paper mills as we focus on the activities of International Publisher and the papers brokered through 123mi.ru.  As part of this project we are curating  a database of all the papers and authorship positions that have been advertised on this website.  Our database consists of roughly 2,353 unique article titles with 8,928 authorship positions.  While the majority of the known paper mill activity has been in the biomedical sciences, our work on just this one paper mill demonstrates that paper mill products have infiltrated multiple scientific disciplines in which career advancement is heavily reliant on academic publications. 

So far, we have identified nearly 200 published articles that may have been brokered through this paper mill and which cross disciplines including (but not limited to) humanities, social sciences, nursing, and education.  We also observe numerous papers on COVID-19 that have been or currently are advertised for sale.  

Our project is far from complete, but we thought it important to report on our methods and preliminary findings via Retraction Watch.  In doing so, we hope to raise awareness of a serious and potentially widespread problem, along with strategies to help detect and possibly prevent fraudulent activities.  

Overview of International Publisher LLC

International Publisher LLC has a very well-established online presence.  Their main site is http://123mi.ru, with several other mirror sites that use the International Publisher LLC branding, like some type of franchise with outposts in different countries.  

• http://buy-sell-article.com/
• http://publisher-international.com/
• http://publisher-asia.com/
• http://publisher-europe.com/
• http://publishinghouseasia.com/
• http://publisher-shanghai.com/
• http://publisher-usa.com/
• http://publisher-science.com/
• http://publisher-russia.com/

In 2019, Clarivate’s Web of Science Group examined the site and found 344 articles were available for sale.  Clarivate sent a cease-and-desist letter to International Publisher.  Retraction Watch followed up, asking for a comment.  A person who identifies as Ksenia Badziun responded: 

Regarding the quantity of the manuscripts we have published, I want to confirm that it grows every time. It could be a pleasure for me to show all the list of the manuscripts we published, but due to the policy of our company and contracts between the authors/publishers and our company, I simply cannot do it. On the other hand, I want to inform that we have our own system and program with all the records and story on each particular manuscript sent to us from the authors. The access to our program is provided to some of the editors-in-chief, publishers and other our Partners.

Ksenia Badziun has a LinkedIn page that identifies her as the Chief Editor of International Publisher.    

We did not discover this paper mill.  A simple Google search of their website (http://123mi.ru) provides dozens of links to articles and blog posts describing this paper mill.  Using a browser plug-in to translate the text from Russian to English, you can easily review hundreds of papers and authorship positions that are currently available for sale.   

As papers are sold their titles are taken offline, and new titles quickly appear.  These titles likely come from a range of sources including direct solicitation of established researchers. For example, an ethics presentation by Ronald Gilman, PhD (an experimental nuclear physicist at Rutgers University), displayed  an email inquiry from Badziun, who asked Professor Gilman:  

I suppose, that you have some manuscripts that were sent to the journals but not accepted yet.  In this case, you might be interested to add a co-author to such a manuscript to get some profit.

Professor Gilman provided us with an original copy of the email, which confirmed that International Publisher is actively involved in brokering authorship positions.   

Curating data on fraudulent activity

One of the challenges of investigating this paper mill is that, as the authorship positions for a given paper are sold, the paper is removed from the website.  Our interest is identifying the thousands of articles that were previously made available for sale and may now be in the scientific literature, but  on the website we can only see paper titles that are currently available for sale. We then discovered that the contracts for each authorship position for all the papers that have been advertised on 123mi.ru are on their web server, even though they do not have links that make them directly accessible through their website.  Thus, we scraped the data using a script in Python.  This process involves crawling through thousands of links and making copies of the contracts that are in HTML format.  We parsed the HTML to extract the contract details, which were saved in a format that can be easily analyzed.  Using these procedures, we obtained contracts for  8,928 authorship positions that were associated with 2,353 unique papers.  These contracts contained the paper titles, target indexing (Scopus or Web of Science), cost of the authorship position, and target publication date.  We note that these contracts did not include any additional information, such as availability dates or names.  

Among these paper titles, 961 articles are currently available for sale, suggesting the remaining 1,392 papers may be in the scientific literature or, alternatively, could not be sold.  We sought to locate published papers which we could link back to a sold paper with a reasonable degree of certainty.  We did this by using the paper titles as initial search terms and seeking matches—both exact and close matches, since titles are sometimes edited during the review process and whatever was advertised could have changed before going into print. An obvious problem with this search strategy is coincidental matches; some areas of study are highly nuanced, so titles with a close or even exact match that were not produced by the paper mill are entirely plausible. To solve this problem, we used matching titles as only one of many different indicators in our search for fraudulent papers.  

Once we had a tentative list of published articles, we investigated whether the authors had a background in the paper’s topic area and/or had a history of working together. We also considered whether the collaborations were multi-institutional and/or multi-national and whether the corresponding authors used non-institutional email addresses (e.g., Gmail, Hotmail, Yahoo, QQ).  Non-institutional emails allow the paper mill to impersonate an author to manage submissions and communicate directly with anybody who raises questions.  We also looked at the outlet for publication. Articles published in open-access journals, especially those with high volumes of output, were considered as potential targets of fraud.  In sum, we used professional judgment—particularly what we know from our own experiences in academic publishing as well as our knowledge of paper mill publications—to flag potentially fraudulent papers. However, we also required the presence of one additional strict criterion in our flagging procedures: temporal precedence.

Temporal precedence

To meet a high evidentiary standard of potential fraud, we must establish that the paper in question was made available for purchase before the article was published—that is, we need evidence of temporal precedence. Otherwise, somebody could copy a title from an article and make it available for sale.  This seems to be the case with Teziran, which is another research paper mill that was reported by Retraction Watch that appears to be reselling published articles.  We wanted to ensure our claims are based not only on the indicators we describe, but also on temporal precedence.  Since the contracts we scraped did not have a timestamp indicating when the paper title was made available for purchase, we relied on the Internet Archive to establish temporal precedence.  

The Internet Archive is a nonprofit digital library that takes historical snapshots of the World Wide Web, which are publicly available through their service called the Wayback Machine.  Over 100 snapshots of the 123mi.ru are currently available on the Wayback Machine, allowing most of their online activity to be recreated from 2016 to the present. The records indicate that 123mi.ru did not start listing paper titles until late 2018, when they introduced a new site layout with a menu of options for buying and selling papers. 

The Wayback Machine is a key data source for a couple of reasons.  For temporal precedence, we can look at the article’s publication date (and the submission date, if available) and establish whether the paper was previously made available for sale. For example, if a paper was made available for purchase on 123mi.ru before the first online publication in a journal, we have evidence of temporal precedence. Additionally, each Wayback Machine snapshot can be scraped in the same manner that was scraped from the 123mi.ru web server.  That means that even if these contracts are removed from the web server, most of the data can be independently reconstructed from the Wayback Machine.  Unfortunately, because the Wayback Machine does not take snapshots at fixed intervals, some data may not be recoverable. 

Preliminary findings

So far, we have flagged and are actively investigating nearly 200 published papers that may have been sold by International Publisher.  We plan to release our database of published paper titles that have likely been obtained from International Publisher but will not make that database publicly available until all individual authors have had the opportunity to comment on the evidence we have compiled. [Update, 4/1/22: It is now available.] Instead, we provide two case examples to highlight the nature of the potentially fraudulent papers we have identified. The authors and publishers referenced in these examples were provided with an opportunity to comment.  

Case example 1: International Journal of Emerging Technologies in Learning

As we were scanning the literature trying to find potential matches, we found a  Russian freelance site known as freelancehunt that was seeking a writer for one of the titles we scraped from the mi123.ru website.  The freelance posting indicated that the “authors of the article” were from Beihang University and Russian State Vocational Pedagogical University.  

This discovery is significant.  First, we found an article published in the International Journal of Emerging Technologies (iJET) with a near-identical title.  Two of the authors were affiliated with the universities noted in the freelancehunt advertisement.  We also established temporal precedence with the Wayback Machine. We therefore have multiple indicators of potentially fraudulent publishing.  Moreover, the freelance post reveals that International Publisher not only brokers authorship positions but also provides ghostwriting services.  

We then found more published articles in iJET, which led us to do a more thorough review of this journal.  We identified a total of 29 papers published by iJET within the last two years that have exact or close matches with titles scraped from 123mi.ru.  As we were establishing temporal precedence with the Wayback Machine, we discovered a highly unusual note from a June 29, 2020 snapshot.  The translated version of the advertisement is shown below:

This advertisement indicates that this article was scheduled for publication as part of a collection of 10 papers.  Because the advertisement includes the IDs of other paper titles (#1081 – #1090), we were able to easily link those to our collection of contracts obtained from the 123mi.ru web server.  Nine of the 10 articles listed in this advertisement were published in the iJET, Volume 16(2), 2021.  As this journal publishes two volumes every month, the release of this issue coincides with the advertisement.  Moreover, among the nine published articles, eight articles had at least one author affiliated with a specific medical institution.  Taken together, these findings raised many questions.  So, our next steps involved contacting the authors of the 29 papers we flagged in iJET, along with the editors.  

iJET:  Author responses

In contacting the corresponding authors of these 29 papers, we received responses from three authors.  The three authors provided different comments and explanations of the evidence.  Because we are unable to confirm the person with whom we were communicating, we are not releasing their names but have provided the editors of Retraction Watch with copies of our correspondence for verification.    

One author insisted that the paper is, in fact, his own work and has no knowledge of the matter and is likely a coincidence.  After this author was provided with our evidence of a second article, he remarked: 

I am very surprised, why the title of my article is the same as this website. It may be because before I submit the article, I will check the repetition rate of this article on the duplicate check website. When I checked the article, the hacker stole my article, so our article title will be the same.  I think this is very possible.

We then discovered a third potentially fraudulent article that was in an entirely different field with different co-authors.  This person blocked our emails, preventing us from obtaining any further comments.  The response from another author was longer and described the difficulties of publishing research in an English language journal and the difficult financial circumstances he is facing.  However, this person vehemently denied any involvement with 123mi.ru:  

I really know nothing about the website you said. My first perception is that there may be colleagues or competitors who slander me or want to amuse me. In order to get a promotion faster than me, some competitors will do anything (such as whether they will steal my paper U disk, publish it on the web page and set up a game to frame me?) we can’t know that the hearts of competitors are dangerous. 

The third respondent also denied our involvement with the website.

It seems to me that your newsletter is a fraud and, possibly, an attempt to steal personal data, which I will report to the advising services.

This person also directed us to their research profile to establish that the publication is legitimate.  We followed up with this person with the historical snapshot and link to the Internet Archive showing that a paper with a near identical title was made available for sale before the publication was online.  We did not receive a response.   

iJET:  Editor’s response

On Aug. 3, 2021, we provided the editor-in-chief of iJET, Dominik May, PhD, preliminary evidence of 19 potentially fraudulent publications that were published in the journal.  Dr. May responded on Aug. 5, 2021, and cc’d the executive editor, Michael Auer, PhD, on the communication.  Drs. Auer and May are the president and vice president, respectively, of the International Association of Online Engineering, which manages seven open-access journals, including iJET.    

Dr. May promptly replied, indicating, “we did not know about such fraudulent activities and are pretty much shocked about it.”  We exchanged numerous emails with Dr. May over the next two months discussing additional evidence that emerged.  On Sept. 27, 2021, Dr. May provided the following official comment:  

For over 15 years, the interdisciplinary International Journal of Emerging Technologies in Learning (iJET) aims to focus on the exchange of relevant trends and research results as well as the presentation of practical experiences gained while developing and testing elements of technology-enhanced learning. So it aims to bridge the gap between pure academic research journals and more practical publications. In this field, the journal’s integrity is our focal concern to ensure that both authors and readers can trust and learn from the work published in our journal. When we were contacted by Dr. Perron’s team and learned about the working results concerning a professional paper mill undermining good scientific practice, we were at the same time astonished about the fraudulent procedure and thankful for getting to know about it. The effort and dedication to detail Dr. Perron and his team put into the investigation is invaluable for us as a journal and goes beyond the realms of possibilities in terms of background checks we have as editors. However, each of the manuscripts published in our journal undergoes a double-blind peer-review process and goes through a process of three editorial checks. The fact that it was still possible to place potentially fraudulent papers in the journal, deeply upsets us. We as the journal leadership can only condemn such procedures and dissociate ourselves from any person or entity, which is part of it! However, the revelations have made us rethink our internal process substantially. In reaction to the unveiled procedures, we immediately informed our editorial board members and shared a list with potential hints to identify further fraudulent manuscripts. Furthermore, we checked every submission in the current review or editorial process and rejected manuscripts matching these hints. To prevent any further critical submissions and address fraud, the additional steps we are taking include but are not limited to run internal workshops with editors and reviewers to detect fraudulent submissions, develop a blacklist of individuals and institutions that are clearly connected to the paper mill, and put a particular focus on checking authors groups’ credentials. Following these steps, we are very positive to detect further fraudulent submissions and ensure our journal’s integrity.

Case example 2:  Energies

Our second example involves a paper title advertised as “Financial Audit Methodology of Energy Corporations.”  After performing a keyword search, we flagged an article published in the MDPI journal Energies. The title was a close match, but not exact: “Impact of Non-Financial Factors on the Effectiveness of Audits in Energy Companies.” A few indicators of potential fraud were observed, such as the article’s being published in an open-access journal and no history of collaboration between the lead author and coauthors located in other countries. However, the strongest evidence came from the Wayback Machine.  Specifically, the Wayback Machine shows that this article was advertised on mi123.ru as early as April 29, 2020, with an abstract that has a remarkably close match with the article that was first received by Energies on Nov. 7, 2020.  We reached out to all the authors of this paper on Sept. 20, 2021, asking for an official comment.  To date, we have not received a reply.  

In addition to this article, we identified five more MDPI articles that have potential links to International Publisher.  The collection of evidence for these papers was submitted to MDPI on Aug. 27, 2021, for review and comment. 

Damaris Critchlow, the head of publication ethics at MDPI, acknowledged receipt and arranged an online meeting on Sept. 21, 2021, with a member of our research team (BEP). In this meeting, Critchlow acknowledged that MDPI has been working actively to address fraudulent activity by paper mills specifically in the biomedical sciences.  Critchlow was “surprised and disturbed” to see fraudulent activity in other areas of science.  She stated that looking at only a single paper in isolation makes fraudulent activity difficult to detect; but, when looking across papers, the indicators of fraud are more apparent.

Regarding the articles we submitted to MDPI, Critchlow agreed that they do have indicators of potential fraud and the collaborations among the authors were very peculiar.  Critchlow stated that MDPI is actively investigating the cases submitted by our team and exploring ways to address paper mill activity more broadly.

Lessons learned

Our process of curating data from 123mi.ru and searching for fraudulent paper titles is described linearly. However, our searches were highly iterative and relied on a variety of strategies.  We uncovered different types of evidence of fraud that were unexpected and served as further indicators to locate other potentially fraudulent papers.  Some of the data we extracted from the contracts also included nuggets of information that helped us understand the business of scientific misconduct, which is essential for detection and prevention.  

Our evidence reveals a diverse set of fraudulent strategies that International Publisher is using to promote fraudulent publications. These include late-game authorships, ghost-writing services, and fake peer-reviews.  We also observed dozens of advertisements of articles sold that are associated with “special issues,” which is a potential mechanism of academic fraud that has already reared its head and deserves further investigation..  Our claims are based on a collection of indicators based on a collection of paper titles obtained from the mi123.ru webserver, in addition to the review of “services” brazenly described on the mi123mi.ru website.  We want to emphasize that at this time we have only indirect and circumstantial evidence for potential fraud.  As we regard scientific knowledge as a public good, we think raising awareness of these instances is critical to ensuring scientific integrity.  

Based on our findings thus far, we think the majority of paper mill publications are primarily intended to satisfy a career requirement rather than making a scientific splash. Many of the publications we have flagged as potentially fraudulent are published in open-access journals that have received the predatory journal moniker in the scientific blogosphere. However, some scholars have very specific institutional requirements for promotion, such as publishing with journals included in the Science Citation Index or the Social Science Citation Index.  

Most of the potential fraud in our investigation involves authors at Chinese and Russian institutions. These institutions typically have onerous publication requirements in English-language journals for obtaining degrees or promotions, although the Chinese government has recently banned such practices.  Many of these scholars do not have the requisite research preparation, financial resources, time, and English language skills to meet the publishing demands of the institution.  Thus, for these scholars, they may be relying on the paper mills as a career survival strategy, or what they consider to be a reasonable response to an unreasonable requirement.  From this perspective, we consider these institutions to be culpable in promoting these fraudulent practices.  

Understanding paper mill fraud is an important step toward detection and prevention.  Unfortunately, academic publishing is highly decentralized, and efforts to uncover fraud require a significant investment of time, technical skills, and content expertise.  Thus, the most effective responses to this problem are likely to come from collaborative activities and efforts within a scientific discipline.  Journal editors and reviewers are perhaps the first line of defense against paper mill fraud.  Editors should take an active role in educating reviewers on the problem of paper mills while establishing clear and visible policies on the journal’s website, including the following:  

• Changes in authorship after submission must receive the editor’s approval and be made only under exceptional circumstances; 
• Corresponding authors who indicate an institutional affiliation must be verified by the author or supervisor using an email address from that institution; 
• All papers must be reviewed and approved by at least one reviewer that has a formal affiliation with the journal’s reviewer board or pool; 
• Authors must include a statement of their contributions, which is published with the final article; and
• Each author  must sign a statement that their submission has met all ethical standards of the journal.

Since we commenced our investigation, the ostensibly complete collection of contracts were removed from the 123mi.ru web server.  The only contracts that are accessible from the web server are for articles that are currently available for sale.  However, the complete collection of data has been shared with Retraction Watch for fact checking.  We are currently reviewing the historical snapshots on the Internet Archive to obtain persistent links to these data.  We will continue investigating potentially fraudulent publications and anticipate releasing the curated data from 123mi.ru web server and our computer code in  an open-access format.  

Brian E. Perron, PhD, is a professor of social work at the University of Michigan. Oliver T. Hiltz-Perron is a student at Community High School in Ann Arbor, Michigan. Bryan G. Victor, PhD, is an assistant professor of social work at Wayne State University.

Like Retraction Watch? You can make a one-time tax-deductible contribution by PayPal or by Square, or a monthly tax-deductible donation by Paypal to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].