About these ads

Retraction Watch

Tracking retractions as a window into the scientific process

No more scientific Lake Wobegon: After criticism, publisher adds a “reject” option for peer reviewers

with 6 comments

doveIf you know Prairie Home Companion, you that that in fictional Lake Wobegon, “all the women are strong, all the men are good looking, and all the children are above average.”

That’s a bit like what Harvard’s Nir Eyal found when he was asked to review a paper for Dove Medical Press. Here’s what he saw when he looked at Dove’s peer reviewer form:

dove beforeThe worst possible rating — a 9 — was that “Paper has major flaws and requires considerable modification to make it acceptable.”

At most other journals, some version of “reject” would be available. As Eyal noted:

Failing to even offer peer reviewers the option of rejecting a paper mocks the peer review system. It is a brazen violation of basic, definitive ethical norms in scientific review.

Tim Hill, publisher of Dove Medical Press — which was expelled from the Open Access Scholarly Publishers Association (OASPA) last fall following the John Bohannon sting published in Science — considered Eyal’s comments, and made a change. The new peer review form reads:

dove after

Hill tells Retraction Watch:

I should explain that we have operated since we started publishing journals on the basis of a minimum of two sets of peer-reviewer comments for each paper. Narrative comments plus the numeric rating and any Conflicts of Interest disclosures are supplied to each journal’s Editor-in-Chief. The Editor-in-Chief then evaluates the reviewer responses along with their own reading of the paper and they then decide what happens to each paper.

To help the reviewers and Editors-in-Chief we have a numeric rating scale on each reviewer response form for all our journals. These are intended to give a quick ranking of the paper that supplements the narrative as a help to the Editor-in-Chief in making their decisions. We put a brief legend on a score of 1, 5, and 9 to give the reviewers some idea of what the scale means.

We have never discouraged our reviewers from rejecting papers. When a peer-reviewer contacted me recently I was interested to receive his comments as it was the first time that anyone had raised such an issue. We are always eager to improve how our processes operate and the peer-reviewer had identified something that was not previously considered. We wanted to improve our process and make the numeric scale as straightforward as possible for users.

When the Bohannon scam paper occurred and we were made aware of it we immediately began to modify our internal processes to prevent anything similar happening in future. These changes were in place well before we heard of the OASPA decision. Our experience with the Bohannon paper taught us that we need to continually improve and adapt our systems and continue to keep them as up-to-date and secure as possible. We think that adopting the recent peer-reviewer suggestion may avoid anyone else taking the view on that numeric scale that he had.

About these ads

Written by Ivan Oransky

February 12, 2014 at 11:33 am

6 Responses

Subscribe to comments with RSS.

  1. I’m not sure that we need a “reject” option… assuming the paper fits the broad aims of the journal and didn’t violate any explicit rules (e.g. is not plagiarism, all authors contributed and agreed to submission, the authors own the data, no fraud) the old option 9 seems just find to me. Specific further comments could include: prior to accepting this publication, 1) all experiments need to be completely redone in line with scientific norms 2) appropriate controls need to be performed 3) a discussion merited by the results should be included 4) more analysis needs to be done 5) or even…. experiments that would make this finding somewhat interesting to someone in the world should be performed before publishing this paper.

    Basically, as long as the comments can say “You really need to start over from scratch…. but we aren’t rejecting you per se” I see no reason why “Paper has major flaws and requires considerable modification to make it acceptable” is not OK. I guess the problems are 1) if the author keeps re-submitting without actually addressing the concerns fully it would be a waste of time 2) if the author quits the review processes without the paper being formally rejected, it could be accepted by a less stringent reviewer.

    TL;DR: I don’t know that a review should need to have the option to reject a paper, so long as they are allowed to not endorse its acceptance.

    QAQ

    February 12, 2014 at 3:04 pm

    • The options are just recommendations, but I prefer that these recommendations are aligned with the options that the editor has. So as the editor can reject a manuscript, I would like to be able to explicitly recommend that course of action. The few journals that I have reviewed for are like this; as a reviewer I have had no more than four options: accept without revision, minor revision, major revision/reject with option to resubmit, reject/reject without the option to resubmit. It certainly makes it easier to make the recommendation because you know exactly what these things map to.

      CH

      February 13, 2014 at 8:11 am

      • I guess I meant that, more inline with the PLOS ONE et al. philosophy, since journals have virtually unlimited space to store/share articles, once it gets passed the screening editor as legit (and unless it later violates a rule), I don’t understand why an article ever needs to be rejected, either by an editor or reviewer. Presumably there is always an amount of better experiments and writing that can make an article meet the scientific standards of any journal. Rejection only becomes an issue when a journal needs to select the most “interesting” articles to increase its impact.

        Which journal is more legit?
        The Hypothetical Journal of Excellent Science, where no paper is ever rejected, but only papers with the absolute highest level of rigorous science and thorough discussions are accepted.
        or
        The Journal of Nature, where papers with low citation potential are rejected regardless of quality and papers with high citation potential are given some leeway.

        It matters much more that there is a stringent requirement for acceptance than an option for rejection.

        QAQ

        February 14, 2014 at 12:21 am

  2. Agreed it is nice to have the reviewer summary boxes in line with editors’ options. But the idea that this was a “brazen violation” of norms that “mocks the peer review system” by not allowing reviewers to reject a paper, does not accord with my experience with reviewers, or academics in general, who are an independent lot whose living is made by presenting/arguing their views to the world. Of course reviewers will give their honest assessment without holding back (sometimes one wishes they would hold back just a little! or at least be more polite about it), regardless of the wording of some button option. To me, the phrase “major flaws and requires considerable modification to make it acceptable” is more or less just a polite way of phrasing “sorry Bub, doesn’t pass muster.” (Is Dove Publishers England-based, by any chance? UK form tends to more polite, less-blunt wording, imho.) That doesn’t mean they actually have lower standards for what makes it into their journals (I have no idea what their standards are.)

    Karen Wynn

    February 13, 2014 at 9:13 am

  3. if I can be a little blunt here…of course Dove Press has not a “reject” button..it is a potentially predatory open access publisher! as Beall says in his 2012 list: “Dove Press
    This New Zealand-based medical publisher boasts high-quality
    appearing journals and articles, yet it demands a very high author fee for
    publishing articles. Its fleet of journals is large, bringing into question how it
    can properly fulfill its promise to quickly deliver an acceptance decision on
    submitted articles. ” (here is the link: http://carbon.ucdenver.edu/~jbeall/Beall%27s%20List%20of%20Predatory,%20Open-Access%20Publishers%202012.pdf

    so…this is why is quick…

    fnord42

    February 13, 2014 at 9:39 am

  4. I recently reviewed a paper for some reasonably prestigious journal that did not in fact have any options for the reviewer to choose. It just asked for the reviewer’s comments on the article but not for an opinion on the final disposition of the article. In fact, I wonder if BMJ actually asks for the reviewer’s opinion as to acceptance or rejection or whether that is decided by the editorial group.

    CarolynS

    February 13, 2014 at 10:30 am


We welcome comments. Please read our comments policy at http://retractionwatch.wordpress.com/the-retraction-watch-faq/ and leave your comment below.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 33,666 other followers

%d bloggers like this: