Scientists are always pressed for time; still, Raphael Didham of the University Western Australia was surprised when he fell upon a group of early career scientists using a spreadsheet formula to calculate whether one was obligated to accept an invitation to review a paper, based on how many manuscripts he’d submitted for review. “I recall that sharp moment of clarity that you sometimes get when you look up from the keyboard and realise the world you (thought you) knew had changed forever,” Didham and his colleagues write in a recent editorial in Insect Conservation and Diversity. We spoke with Didham about how to convince scientists that peer reviewing is a benefit to their careers, not a burden.
Retraction Watch: You talk about the current problem of “zero-sum” reviewing. Could you define that in the context of the scientific peer review system?
Raphael Didham: Peer review is one of the gold standards for evaluating the quality and rigor of research prior to publication in a scientific journal. Most ‘peers’ in the peer review process are other scientists in the same academic field, who by and large carry out this work on a voluntary basis. The motivation for doing this varies among individuals, but for most people it is a balance between the ‘peer esteem’ in being invited to be an expert reviewer, and the ‘obligation’ felt in repaying a perceived debt owed to the peer review system because their own scientific papers have benefited from the receipt of anonymous reviews. It is perhaps only natural that voluntary commitments such as this are the first to be dropped as the demands on scientists’ time continue to intensify in academia. Certainly, the increasing difficulty we have in finding willing reviewers is a common point of discussion among journal editors. My perception from speaking to many scientists is that they would like to accept more peer review invitations if they had more time, and many struggle with finding an objective mechanism to justify turning down reviewer invitations while still meeting their perceived obligations. This is where ‘zero-sum’ reviewing comes in.
Many scientists are inherently ‘analytical’ people, and naturally gravitate towards accountable metrics of performance. In a zero-sum game, researchers could quantify the minimum effort required to resolve the ‘reviewer debt’ owed when they publish one of their own papers, simply by calculating the number of reviews received per paper (k) divided by the number of coauthors per paper (n). This simple formula, Sk/n, seems disarmingly ‘objective’ and superficially ‘fair’ in apportioning obligation – but actually holds a number of inherent biases that are having an unduly negative effect on reviewer willingness to review.
RW: Why is “zero-sum” reviewing such a problem in science publishing today?
RD: We believe that a growing philosophy of ‘zero-sum’ reviewing is one of the factors contributing to the increasing difficulty in finding willing reviewers these days. What this does is create a bottleneck in manuscript processing, and growing delays in the publication of new research. Imagine a scenario in which a large productive research group is producing 20 manuscripts per year, with say 20 co-authors per manuscript. Peer review of this body of work would demand a substantial amount of commitment from peers in the community (at a minimum let’s say 40 peer reviews, if at least two are required per manuscript). Under a zero-sum ethic, however, each co-author might only feel obliged to pay back their personal ‘marginal debt’ of 2 / 20 (reviews per co-author) for each of the 20 manuscripts – i.e. Sk/n = 2.0. So, even though some of the members of this research group might be publishing 20 papers per year, they might only feel the need to pay back two peer reviews to the community. In practice, what this means for journal editors is that increasing numbers of potential reviewers feel they are justified in declining review invitations because their personal obligation is perceived to be low. Therefore, journal editors must send out many more invitations in order to secure at least two willing reviewers.
RW: How have you seen the effect of larger numbers of co-authors on “zero-sum” reviewing at your own journal?
RD: At Insect Conservation and Diversity, we have noticed a strong directional trend toward increasing numbers of coauthors per paper through time, which is part of a worldwide pattern of ‘hyperauthorship’ in many disciplines. For zero-sum reviewers this would no doubt lower their perceived obligation to pay back an increasingly tiny share of the reviewer debt, and k/n (the average number of reviews per co-author) has declined linearly through time at Insect Conservation and Diversity, from ~1.2 in 2008 to ~0.6 in 2016. Over this same period, the median frequency of reviewers accepting review invitations has dropped noticeably from ~70% in 2008-2013 to ~50% in 2014-2016. More worryingly, an increasing percentage of manuscripts now require excessively large numbers of review invitations (i.e. ≥8) to make sure we receive at least two completed peer reviews, from ~6-8% in 2008-2011 to ~15-20% in 2014-2016. In fact, we have had some manuscripts that have had as many as 15 reviewer invitations turned down, and I know of other journals with even more extreme examples of declining reviewer willingness to review. It is easy to see what a massive bottleneck this could impose on the rapid publication of science.
RW: What can journals do to tackle the problem of a lack of willing reviewers?
RD: We believe that a combination of changes is required in both journal processes and reviewer attitudes, in order to tackle this problem. For journal editors, we need to minimize the disincentives to review, while promoting the incentives. In our editorial, we point out how weak editorial oversight can quickly lead to a reputation for sending out manuscripts of low or mixed quality for review, and this can be a major disincentive to reviewer willingness to accept review assignments. Strong editorial process balances the ‘gatekeeper’ role of good editors in rejecting obviously-flawed work prior to peer review, against the ‘facilitator’ role of good editors in weighing up a consensus of reviewer criticisms and author responses. Strong editorial process also involves making good judgment calls on whether it is absolutely necessary for revised manuscripts to go back to reviewers a second (or third) time for approval. At Insect Conservation and Diversity, for example, we calculate that implementation of stronger editorial process has ‘saved’ approximately 78 peer reviews per 100 manuscript submissions, and potential reviewers can more reliably expect to receive higher quality manuscripts to review.
In terms of incentives to review, many publishers offer various types of rewards schemes, such as free journal subscriptions, discount publication offers, and awards or prizes for high levels of reviewing service. Interestingly, feedback on our editorial has suggested that early-career researchers (in particular) are reluctant to undertake peer review obligations because of the time commitment and perceived ‘lack of credit’ and ‘lack of tangible benefit’ that will help them get a job or improve promotion prospects. These comments seem to revolve around the desire for a numerated accounting of effort that they can use as a measure of peer esteem on their CV. Without this, they would rather decline review invitations and spend the time writing their own manuscripts. Regardless of the pros and cons of these perspectives, one way that some journals are providing this kind of numerated accounting is via ‘publons,’ which tracks and validates peer review effort across journals for registered reviewers (and editors). This is something we are currently discussing at Insect Conservation and Diversity.
RW: What would you like to say to readers who may – consciously or unconsciously – be zero-sum reviewers, to try to convince them to adopt another approach?
RD: For researchers out there, nine-tenths of the battle lies in changing their attitude about peer review invitations from ‘burden’ to ‘benefit’. Readers should reflect on the reasons why they are doing science in the first place, and take a moment to consider the genuine competitive advantage they can gain in their career from receiving an advance preview of new developments in the field before they are even published. Peer review provides a low-cost synthesis of how up to date you yourself are with current literature, a benchmark of comparative performance with other researchers in the field, and the opportunity to shape the conceptual and technical direction of your field through critical feedback. So, instead of just complaining about how much ‘rubbish’ is being published these days, and the ‘failure’ of the peer review system to mitigate this, the answer is to be a more constructive part of the solution. There is also great esteem in being invited to be an expert reviewer in your field, and our experience suggests that the benefits accrue in direct proportion to the quantity and diversity of manuscripts that you review. The senior editors at Insect Conservation and Diversity each handle ~60-70 manuscripts per year, and 10-15 manuscripts each for two or three other journals that each of us is an editor on, plus 30-50 peer reviews per annum for other journals in our disciplines. A quick scan of publons peer review ‘heroes’ suggests that some people are doing many times more than this amount as well. These are perhaps aspirational targets for many early career researchers, but why not take the opportunity to have an important influence on the development of your field – think ‘benefit’ not ‘burden’ next time you receive that review invitation.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
On the one hand, as a postdoc I do enjoy getting to “preview” cool stuff ahead of publication. And of course I respect that my own papers have been substantially improved through peer review. BUT – when journals are all for profit, and I as a researcher have to pay to publish in them, and pay to read them (or my university pays for access), why on earth should I work for them for free?
Completely agree. We pay to publish and pay to access, yet write and review for free? Instead of complaining about lowly-paid post-docs being reticent about reviewing, editors should be thinking about how to re-design the entire system for the internet age.
Not all journals are pay to publish – all the Royal Entomological Society journals are free of page charges, unless you go for Open Access
Because if you want payment, article processing fees (for OA journals) or subscription rates (for non-OA journals) would need to go up…
Every scientist who publishes in a peer reviewed publication has a moral obligation to be a peer reviewer, with or without payment. Consider this situation: DT submits a MS five times and is finally accepted on the fifth round, having been helped by 12 reviewers. DT benefits from the reviews and would not be published without those reviewers. Clearly, DT has an obligation to agree to review 12 MSS. It’s only fair; science cannot survive without a strong sense of community. We all need each other to do our part.
No.
Nowadays, DT’s article’s author list probably has 12 or more authors on it. So in your scenario, each of them is “morally obligated” to review 1 article. No more.
I totally agree with Ken’s argument and that’s (roughly) how I estimate my annual reviewer debt to the scientific community. As a caveat, to me that math only applies to the first and senior author of a paper i.e. the people who will receive the lion’s share of credit.
Where is the morality of the publishers in this situation? I should pay to publish and perform peer-reviews for free?
Cautiously agree until the bit “DT benefits from the reviews and would not be published without those reviewers.”
I cannot count the number of times reviewers have suggested the paper THEY think it should be as opposed to the actual results. In my opinion, the quality of peer reviewing has dropped in the last 20 years and this is affecting outcomes.
I tend to agree with Ken, community effort is needed to support publication costs. However, this makes sense in a world where universities pay for subscriptions, researchers offer voluntary refereeing service, publishers perhaps offer reduced rates to developing countries, etc.. In case of authors paying for publishing, this is not sustainable and APCs should go to parties actively partecipating in obtaining the final product: a good paper in a good journals. In other words, referees should be paid, e.g. from APCs.
I wonder whether the people who use this logic to calculate their obligation to review also applied it equally to non-profit, open access journals. I work on a few myself and although some people do decline requests to peer review papers, the vast majority accept the task. The most common times that they don’t are around the typically busy times of the year (beginning and end of term).
I do know that a lot of people now will not review for large commercial publishers simply because they want to force the publishers to re-think their business model in a way that benefits us as academics (or rather, stops taking advantage of us). This makes me wonder how many of the people who tell them “I’m too busy.” are actually too busy or are simply being polite in turning down the request.
How about refusing to review for journals that desk reject your manuscripts?
So, refuse to review for journals that pre-screen, and only review for ones that don’t? Seems likely to lead to reviewing more worthless manuscripts and wasting more time. (Let’s not kid ourselves that *our* precious manuscripts are never appropriately desk-rejected, okay? Speaking as someone whose last paper accumulated two of those.)
Ask yourself: what am I trying to get the journal to do?
I’m not suggesting to refuse all review requests from such journals, only in the appropriate ratio relative to the amount of time wasted satisfying arbitrary and onerous submission requirements from such journals.
I guess I sort of do this too, but I also take into account the time spent reviewing all of the papers that I’ve read, not just the ones I’ve submitted. I’ve clearly benefited from that unpaid effort at no cost to myself, and at this point I probably owe the community several thousand reviews.
The bigger problem I see is an increase in the amount of total and utter CRAP that I’m asked to review. These are papers for which the title alone is enough to make a decision, because the very premise of the work is so fundamentally flawed.
You know the kind of papers… a poorly-defined multi-component extract from an exotic plant, kills a cancer cell line at ridiculous high concentrations, with a couple of crappy western blots of some kinases, and a discussion section where the authors think they’ve cured cancer and should be in clinical trials already.
This poses a new dilemma… do I refuse to review and risk the paper being sent to someone more sympathetic and published anyway, versus accepting the review and rejecting the paper immediately due to aforesaid fundamental flaws?
Maybe the journals should be paying their editors to screen this stuff before choosing to clog my in-box with it. While Nature et al. have taken this approach to the extreme, there has to be something better than the status quo, where it’s a major time suck just to keep logging into journal websites and coming up with excuses and suggestions for alternative reviewers.
I figure I that my debt to the community is 3x more reviews than the number of corresponding-author manuscripts I submit. I do a bit more than that because I still have a hard time saying no when I know that the paper is a perfect fit to my expertise, or when I suspect its problematic and I suspect other reviewers might not take the time to carefully debug it.
Earlier in my career I was saying yes to nearly every review request, and finally realized that all that reviewing was seriously slowing down the writing of my own manuscripts and grant applications. After I started saying no more, the requests and guilt trips died down a bit, and now things are more rational.
Like Paul Brookes said, whether or not to spend time on papers from labs known to be sloppy is a real dilemma. A particularly tricky one: I realized I was enabling a lab of image-manipulators because when I reviewed their work, I reported it to the editors and the authors just sent a “fixed” version back that the journal took over my objections, whereas they had to publicly retract a paper I didn’t review.
So … its a difficult balance. No easy fix. If journals paid reviewers, there would be a new hornet’s nest of conflicts of interest.
This is the key: “Earlier in my career I was saying yes to nearly every review request, and finally realized that all that reviewing was seriously slowing down the writing of my own manuscripts and grant applications. After I started saying no more, the requests and guilt trips died down a bit, and now things are more rational.”
It’s about only having so much time in a day, and being pulled in too many different directions. When you are early career in academia, your priority HAS to be getting your own manuscripts published and your grants funded. Otherwise you won’t be there in the future to help review any number of papers. That’s the bottom line, sorry editors but you need to press the senior people more and give those early career folks a break.
Pay reviewers and you won’t have this problem. I often give the ‘too busy’ excuse, since I am on 100% soft money and spend my time grantwriting, but if there was a honorarium I would be much more likely to find the time.
I admire the community spirit that a lot of people embrace when it comes to peer review, but there is another side to it. If you’re in a very low paying research job that works you so hard you barely have a Saturday off to work on your own publications then it’s tough to accept the idea that you somehow owe the discipline a favor. Indeed, many people who have done peer review as an early career researcher might never actually end up with a stable academic career at all.
Actually, “review heroes” lose twice: once in terms of time invested in other people’s career progression, and again for pointing out problems in the papers of community “leaders” which will get the reviewer effectively ostracized.