Journal stings come in various shapes and sizes. There are the hilarious ones in which authors manage to get papers based on Seinfeld or Star Wars published. There are those that play a role in the culture wars. And then there are some on a massive scale, with statistical analyses.
That’s how we’d describe the latest paper by Ottawa journalologists Kelly Cobey, David Moher and colleagues. We asked Cobey and Moher to answer some questions about the recently posted preprint, “Stress testing journals: a quasi-experimental study of rejection rates of a previously published paper.”
Retraction Watch (RW): What prompted you to do this study?
Kelly Cobey and David Moher (KC and DM): In 2013 John Bohannon conducted a large sting study where he submitted obviously problematic papers to 304 journals and tracked their response. Some of the journals were presumed to be predatory, while others were presumed to be legitimate open access journals. 157/304 submissions were eventually accepted, and acceptances came from both types of journals. Since then there have been many small sting studies showing how easy it is to get problematic work published in predatory journals.
We wanted to build on these studies, and do something more methodologically rigorous. It has been several years since Bohannon’s study. Our study allows us to consider how the scholarly landscape may have changed over time. Similarly, we conceptualized our study as a stress test – did journals, regardless of how they are classified, have sufficient journalology defences in place to withstand inappropriate publication practices.
RW: Describe what you did.
KC and DM: We submitted a paper we previously published in Nature to more than 600 journals from three groups: presumed predatory journals, open access journals, and subscription based journals. Approximately half of the journals we submitted to received the publisher’s formatted version in PDF, while the others received the accepted manuscript version made in Microsoft Word. We tracked journal decisions on the submissions over the period of about 1 month. This study used deception; we lied during the submissions of the article and indicated it was not published previously. We received ethics approval from our research institute and permission to use the Nature paper from Nature.
RW: You posted your protocol in advance, which is of course good scientific practice but means that some journal editors may have been on the lookout for a manuscript including David Moher’s name. Does that concern you?
KC and DM: We did post our protocol in advance, but we embargoed it so that it was not publicly available until the study was complete. This format of registration allowed us to be transparent, but maintain the deception necessary for the study.
RW: We imagine that some critics of Beall’s list may find the choice of it as a source of predatory journals to be problematic. How would you respond?
KC and DM: Beall’s lists (single journal publishers and multiple journal publishers) have many shortcomings. For example, it is unclear to us how he discovered new journals and publishers to consider for his lists. The criteria he used to list journals and publishers were not always transparent. We considered this when designing our study, but there were few other options that were practical. One challenge is that many presumed predatory journals are not indexed. This makes it hard to identify them. The only other list of predatory journals we know of is from Cabell’s; this list is behind a paywall and may suffer from similar shortcomings to Beall’s list. Our work was unfunded, so we didn’t pay to access it.
At the time we conducted our work, there was no consensus definition of what a predatory journal is. We now have a definition, and could use it in future work to assess journals to determine if they meet the definition.
RW: What were your findings? Did any surprise you?
KC and DM: We received correspondence back from 308 journals of the 602 we submitted to. Just 4 of the journals we submitted to accepted our paper, and all of these were suspected predatory journals. Three accepted the word version, and one accepted the PDF version. 13 journals requested a revision of the article (1 presumed predatory, 6 open access journals, 6 subscription-based journals).
We were surprised by just how few journals accepted our paper. The rate of acceptance was much lower than in the earlier Bohannon sting study. We thought that we might see some open access and subscription-based journals accept the paper too, but they did not.
RW: Some 40% of the journals rejected the manuscript because it was out of scope. Do you have any information on whether this was just a way to reject a problematic paper without making an allegation of duplication or plagiarism?
KC and DM: We do not know if editors who rejected our paper for being out of scope also identified issues with plagiarism. If they did, they failed to express these concerns. We think that failure is problematic since journals play an important role in maintaining publication ethics. COPE provides guidance on what editors should do when the suspect plagiarism in a submission; journals have a responsibility to act.
RW: You write that “a very small number of editors contacted the institution of the corresponding author to express their concerns about the submission.” What happened in these cases?
KC and DM: Our study received ethical approval from our institution and Nature gave us permission to use our paper for the purposes of the study. We also informed the Head of our Program and the CEO of our institution that the work was happening. When editors contacted the institution expressing concern about the paper, whomever received the correspondence at our institution did not respond until the end of the study period. This was done to ensure the validity of the study – we did not want word to spread about the study within the editorial community. All journals we submitted to were sent a debrief form at the end of the study. The chair of the ethics committee was the conduit for responding regardless of whether the compliant was made to Dr. Moher’s research institute CEO or university faculty (Dean of Medicine).
RW: You write that your “findings suggest that all three types of journals may not have adequate safeguards in place to recognize and act on plagiarism or duplicate submissions.” It will likely come as no surprise to hear that we agree. What kinds of safeguards would you recommend?
KC and DM: Journals should apply plagiarism detection software to help address this issue. We recognize however that this tool comes at a cost, and it may not be feasible for some under resourced journals.
Journals should also engage in education. Many editors receive very little training before assuming their roles. The same is true of peer reviewers. Making those involved in the peer review process better informed of publication ethics issues is essential. They also need to be provided with clear guidance in terms of how to act when they suspect an issue.
Adopting open science practices may also help journals ensure that what they are publishing has not been published previously. For example, if all studies were registered and corresponding articles/preprints/materials were digitally linked to registrations, it would be possible to check what work has been published related to the registration.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Two observations:
(1) I find this study absolutely infuriating. Sure, the authors got IRB approval from their institutions. But the cost of this study could have been very high. As an upper bound, let’s imagine that all 600 outlets had sent this piece for review to three reviewers and let’s suppose that each reviewer spends an hour on their review: this was possibly up to 75 days of free work (!) wasted by people who provide reviews, for most part, in good faith. And you can add to this time spent by editors and associate editors.
I’m all for getting predatory journals out of business and improving the peer-review process. But there is a true cost to this kinds of studies. I hope this publication was worth the free work done by PhD students and junior faculty.
(2) The paper they submitted was an article about predatory journals. I can’t imagine why editors wouldn’t perceive it to be a good fit for their journal… I can confidently say that in my field, no journal would even entertain publish something so off topic. So what’s the point?
I might be missing something here, but the authors essentially sent an irrelevant paper that was unlikely to be published, took away time from unsuspecting participants, for a result that barely moves anybody’s priors. How is this reasonable?
I totally agree – mostly because I was one of those reviewers!! I wasted several hours on this paper, then recommended rejection (because of the quality of the article which was fine for a Nature “blog” but was not a quality research article). Only a few weeks later did I realise that the paper had already been published – and because the article I received for review was blinded I informed the editor that it may have been plagiarised (more waste of my time). The editor told me that they had identified the problem and were in contact with the authors – like me they were VERY annoyed by the waste of their time, and the time of their hard-to-obtain reviewers. I find it difficult to respect an ethics committee (or the authors) that allowed this huge waste of reviewer/editor time. The fact that so few journals accepted the article shows how futile the research was. Yes, maybe it was interesting to discover that so few journals are willing to accept rubbish, but really not worth the time (600 journals: 2 reviewers and at least 1 editor per submission… as a journal editor myself I would be furious to receive this article). Sorry for sounding sour, but I have far better things to do with the hours I spent on this article.
This is a very poor study, partly for the reasons noted above by Mic, and partly for the very obvious reason that the submissions are more unethical and more problematic than any acceptance. I looked at Science and Engineering Ethics, and found this in the “Author Ethical Guidelines”:
Maintaining integrity of the research and its presentation is helped by following the rules of good scientific practice, which include*:
* The manuscript should not be submitted to more than one journal for simultaneous consideration.
* The submitted work should be original and should not have been published elsewhere in any form or language (partially or in full), unless the new work concerns an expansion of previous work. (Please provide transparency on the re-use of material to avoid the concerns about text-recycling (‘self-plagiarism’).
It’s clear that this submission violates both of these rules. In publishing, there is an implied code of ethical behavior – that the authors will not try to deceive the journal or reviewers. This code was clearly violated. And, as Mic notes, what’s the point of this example?
KC and DM: We do not know if editors who rejected our paper for being out of scope also identified issues with plagiarism. If they did, they failed to express these concerns. We think that failure is problematic since journals play an important role in maintaining publication ethics. COPE provides guidance on what editors should do when the suspect plagiarism in a submission; journals have a responsibility to act.
The COPE guidelines are irrelevant here, since they expressly exclude the case of self-plagiarism (‘redundancy’). The present sting paper retained the main author’s name so it was not plagiarised stricto sensu.
A perfect example of something that needs to be considered a PUBLIC UTILITY and NATIONALIZED or otherwise made available for general use: plagiarism detection software.
RE: We recognize however that this tool comes at a cost, and it may not be feasible for some under resourced journals.
Sorry, but despite all the marketing to the contrary, there is no such thing as “plagiarism” detection software, only text-matching software that sometimes finds plagiarism but often misses and also marks original text as problematic. See my Nature commentary from March 2019: https://www.nature.com/articles/d41586-019-00893-5 or any of my tests of such software. People project their wishes and desires into software, but there are some things that just aren’t doable with software. In particular, social problems (and plagiarism and predatory journals I consider to be social problems) cannot be solved by the use of software.
As one of the editors that received the paper and rejected it I must say that I do not like the whole thing. The paper was from the start very strange one: it had the title “Stop this waste of people, animals and money”. The sold author was David Moher but he used a gmail address: [email protected] He then confirmed he is the author but we decided then to just reject the paper and not contact his faculty since it was clear that the whole thing was just a stupid game.
Also, the whole story was very bizarre, we were not sure if the author is just weird or serious. We ended up excluding him from further publishing in our journal.
Without doubt it was a serious waste of the time of at least 3 people including myself.
Did “mic”, Paul Alan Thompson or Ari Waisman even read the authors’ preprint? The answers to all of their questions are in there.
“mic” observes that “the authors essentially sent an irrelevant paper that was unlikely to be published”, yet this very submission was published (in Nature no less) in 2017.
Paul Alan Thompson notes that “It’s clear that this submission violates both of these rules. [simultaneous submission and original content]” That was the point of this exercise, as discussed by the authors in their introduction and elsewhere in their paper.
Ari Waisman, a self-identified editor (of a Springer Nature journal no less) reports that “we decided then to just reject the paper and not contact his faculty since it was clear that the whole thing was just a stupid game.” How is this any demonstration of responsible editorial oversight, from an Editor-in-Chief no less? Contacting the faculty of an institution whose staff is submitting a previously published paper is part of a responsible editor’s job.
Waisman further exclaims “Without doubt it was a serious waste of the time of at least 3 people including myself.” The whole point of editorial oversight is that when a resubmission of an article is attempted, you are the 3 people whose time should be wasted, so that thousands of researchers, journalists and others do not waste orders of magnitude more time reading illicit articles. If you can not abide such a waste of your time, then please step aside as editor-in-chief.
Furthermore, as a journal in the Springer Nature group, with the following stated policy
https://www.springernature.com/gp/policies/editorial-policies
“Plagiarism
We believe that Springer Nature has a responsibility to investigate any instances of plagiarism that it detects.
We achieve this by routinely using the Crossref Similarity Check powered by iThenticate on submitted manuscripts, and having clear guidelines in place for editors. Where a paper is suspected or discovered to be based on plagiarism, the editor must take prompt steps to investigate and to notify readers. The paper will be corrected or retracted depending on the severity of the case.”
why wasn’t the submitted article checked with the available “Crossref Similarity Check” by the editorial and review staff at the Journal of Molecular Medicine?
Like “mic”, I too find this paper “absolutely infuriating”, precisely because such studies are still necessary. I can’t wait until the day such studies are no longer needed, but we are not there yet.
I have wasted many hours (hundreds at a minimum, most of them my personal time) during the course of my research because of dozens of shoddy papers, many published by apparently reputable journals, that I have reviewed for inclusion as references related to topics for papers I have co-authored. I have uncovered sham statistical analyses and manipulated images, which I have clearly documented and submitted in writing under my name (no anonymous submissions), nearly all to no avail. Some of the very editors to whom I have submitted such findings have themselves been authors of papers with sham statistical analyses and manipulated images. The foxes are “guarding” the hen house.
I thank the group at the Ottawa hospital Research Institute for undertaking this study. I do take heart that the publication rate in such an exercise appears far lower than previous attempts, I’ll move onward with the hope that we are making progress in the effort to reclaim a reasonable space for the publication of scientific findings. I note also with optimism that the authors first submitted their paper to a pre-publication archive, unfortunately a most necessary tool in the current effort to clean up the scientific publication arena where for-profit journal businesses are performing so documentably poorly.
I’ll just reply here to the points addressed to me.
““mic” observes that “the authors essentially sent an irrelevant paper that was unlikely to be published”, yet this very submission was published (in Nature no less) in 2017.”
– A paper on this topic would never be accepted by a flagship journal in any major discipline. The original paper was of interest to a narrow set of journals: mainstream outlets (Nature/Science/PNAS) or journals on the philosophy of science (and perhaps sociology). But virtually no other journal would have been interested *even if this had been an honest submission*. This is not a criticism of the quality of the scholarship or the authors. It’s just that the theme of this paper would be a poor fit for most journals out there. The baseline expectation should have been that this paper would never even be sent out for review.
“Like “mic”, I too find this paper “absolutely infuriating”, precisely because such studies are still necessary. I can’t wait until the day such studies are no longer needed, but we are not there yet.”
– What did this study contribute? This article won’t change anybody’s priors. However, it came at a real cost.
Let me put it another way: would you be okay giving away 90 minutes of your time on false grounds? This study was the equivalent of asking potentially hundreds of RAs to do free work for them while letting them believe that they were doing something useful.
All this for what? To tell us that predatory journals are bad and that journals aren’t perfectly well prepared to deal with plagiarism?
For my part, this kind of stunts just encourage me to get out of the reviewing business altogether. Currently I’m doing free work (about 35 articles per year, i.e. about 40-50 hours – almost a week!). Adding to this reviews for people trying to test the system for no discernible benefits? Not interested, sorry. I hope reviewers who took part in this sent a bill to U Ottawa for the time they spent on it.
This all may seem harsh, given that I genuinely believe that the authors have good intentions. There is much wrong in current academic research. However, I find it extremely annoying that the way they used to show this was to rely on the goodwill of the few actors in the process that are not to blame.
The first “real” experiment of this kind was not Bohannon’s but Peters et Ceci’s… in 1982. It led to the same kind of discussions about its validity, scope and interest, though organized in the same issue where it was published (by S. Harnad, then BBS editor). Of course, the question of self-plagiarism, COPE norms, IRBs, reviewers waste of time and so-called predatory journals are new, but the rest remains the same.
All scholars on peer review have been knowing for some time that it was never made to find cheaters, so “experiments” like this, even more when taking into consideration the short time frame and the topic of the paper, won’t change a thing.
So beyond the supposed “outcomes” of a study that wished it was “more methodologically rigorous”, why are we still thinking PR as an industrial-like process of manuscript management, where automatic warnings would be triggered if someone tries to lure journals editors?
I would treat obnoxious submissions like this with a “Corresponding author could use a horsewhipping or two…” recommendation.
Bad science. In fact this study has been so badly conducted that it could almost be used as a type-specimen of how not to carry out an experiment.
(1) If you want to ask the question “are journals resistant to plagiarism?” you have to make your plagiarism realistic. Not many people resubmit a complete Nature paper word-identical. Most sensible plagiarists reuse (parts of) a few figures as figures in a different paper, together with smatterings of (nearly) identical text, all of it highly relevant to the journal. This is much harder to spot. This study used a very unrealistic plagiarism, and therefore answers the wrong question. It’s like a study (which really happened, many years ago) that aimed to look at whether spiders put bits of leaf in their webs to avoid bird-damage, but did so by putting white paper crosses in webs. Unfortunately the only available interpretation was that birds don’t fly into white paper crosses – it failed even to show that birds avoid leaves.
(2) The authors have no idea whatsoever what would have happened had the paper not been plagiarised, and unless they can demonstrate that the journals would have rejected it differently had it not been a copy, they can’t say anything about the effectiveness of the journals’ antiplagiarism strategy. Anti-plagiarism (OK, text-matching…) software also gives false positives, so it’s possible some journals would have complained at a fresh paper. But more subtly, this would have given some handle on whether the journals who rejected for other reasons were actually motivated by concerns about plagiarism, but didn’t want to get into a debate. It’s so easy to return an article as out-of-scope (especially when it is! why waste time testing for plagiarism when you wouldn’t accept the paper anyway?). So the authors should have written a fresh, closely-related paper and submitted it to half the journals, selected randomly.
(3) Similarly, we don’t know whether the half of journals who never replied, failed to reply because their editors had taken one look at this single-author, out-of-scope item and thought “product of a crank. What do I do?” and then never got round to doing anything at all (that could easily have happened were I editor). The ideal experiment probably ought to have looked at the responses to genuine, relevant, non-cranky papers too.