Scientists are always pressed for time; still, Raphael Didham of the University Western Australia was surprised when he fell upon a group of early career scientists using a spreadsheet formula to calculate whether one was obligated to accept an invitation to review a paper, based on how many manuscripts he’d submitted for review. “I recall that sharp moment of clarity that you sometimes get when you look up from the keyboard and realise the world you (thought you) knew had changed forever,” Didham and his colleagues write in a recent editorial in Insect Conservation and Diversity. We spoke with Didham about how to convince scientists that peer reviewing is a benefit to their careers, not a burden.
Retraction Watch: You talk about the current problem of “zero-sum” reviewing. Could you define that in the context of the scientific peer review system?
Raphael Didham: Peer review is one of the gold standards for evaluating the quality and rigor of research prior to publication in a scientific journal. Most ‘peers’ in the peer review process are other scientists in the same academic field, who by and large carry out this work on a voluntary basis. The motivation for doing this varies among individuals, but for most people it is a balance between the ‘peer esteem’ in being invited to be an expert reviewer, and the ‘obligation’ felt in repaying a perceived debt owed to the peer review system because their own scientific papers have benefited from the receipt of anonymous reviews. It is perhaps only natural that voluntary commitments such as this are the first to be dropped as the demands on scientists’ time continue to intensify in academia. Certainly, the increasing difficulty we have in finding willing reviewers is a common point of discussion among journal editors. My perception from speaking to many scientists is that they would like to accept more peer review invitations if they had more time, and many struggle with finding an objective mechanism to justify turning down reviewer invitations while still meeting their perceived obligations. This is where ‘zero-sum’ reviewing comes in.
Many scientists are inherently ‘analytical’ people, and naturally gravitate towards accountable metrics of performance. In a zero-sum game, researchers could quantify the minimum effort required to resolve the ‘reviewer debt’ owed when they publish one of their own papers, simply by calculating the number of reviews received per paper (k) divided by the number of coauthors per paper (n). This simple formula, Sk/n, seems disarmingly ‘objective’ and superficially ‘fair’ in apportioning obligation – but actually holds a number of inherent biases that are having an unduly negative effect on reviewer willingness to review.
RW: Why is “zero-sum” reviewing such a problem in science publishing today?
RD: We believe that a growing philosophy of ‘zero-sum’ reviewing is one of the factors contributing to the increasing difficulty in finding willing reviewers these days. What this does is create a bottleneck in manuscript processing, and growing delays in the publication of new research. Imagine a scenario in which a large productive research group is producing 20 manuscripts per year, with say 20 co-authors per manuscript. Peer review of this body of work would demand a substantial amount of commitment from peers in the community (at a minimum let’s say 40 peer reviews, if at least two are required per manuscript). Under a zero-sum ethic, however, each co-author might only feel obliged to pay back their personal ‘marginal debt’ of 2 / 20 (reviews per co-author) for each of the 20 manuscripts – i.e. Sk/n = 2.0. So, even though some of the members of this research group might be publishing 20 papers per year, they might only feel the need to pay back two peer reviews to the community. In practice, what this means for journal editors is that increasing numbers of potential reviewers feel they are justified in declining review invitations because their personal obligation is perceived to be low. Therefore, journal editors must send out many more invitations in order to secure at least two willing reviewers.
RW: How have you seen the effect of larger numbers of co-authors on “zero-sum” reviewing at your own journal?
RD: At Insect Conservation and Diversity, we have noticed a strong directional trend toward increasing numbers of coauthors per paper through time, which is part of a worldwide pattern of ‘hyperauthorship’ in many disciplines. For zero-sum reviewers this would no doubt lower their perceived obligation to pay back an increasingly tiny share of the reviewer debt, and k/n (the average number of reviews per co-author) has declined linearly through time at Insect Conservation and Diversity, from ~1.2 in 2008 to ~0.6 in 2016. Over this same period, the median frequency of reviewers accepting review invitations has dropped noticeably from ~70% in 2008-2013 to ~50% in 2014-2016. More worryingly, an increasing percentage of manuscripts now require excessively large numbers of review invitations (i.e. ≥8) to make sure we receive at least two completed peer reviews, from ~6-8% in 2008-2011 to ~15-20% in 2014-2016. In fact, we have had some manuscripts that have had as many as 15 reviewer invitations turned down, and I know of other journals with even more extreme examples of declining reviewer willingness to review. It is easy to see what a massive bottleneck this could impose on the rapid publication of science.
RW: What can journals do to tackle the problem of a lack of willing reviewers?
RD: We believe that a combination of changes is required in both journal processes and reviewer attitudes, in order to tackle this problem. For journal editors, we need to minimize the disincentives to review, while promoting the incentives. In our editorial, we point out how weak editorial oversight can quickly lead to a reputation for sending out manuscripts of low or mixed quality for review, and this can be a major disincentive to reviewer willingness to accept review assignments. Strong editorial process balances the ‘gatekeeper’ role of good editors in rejecting obviously-flawed work prior to peer review, against the ‘facilitator’ role of good editors in weighing up a consensus of reviewer criticisms and author responses. Strong editorial process also involves making good judgment calls on whether it is absolutely necessary for revised manuscripts to go back to reviewers a second (or third) time for approval. At Insect Conservation and Diversity, for example, we calculate that implementation of stronger editorial process has ‘saved’ approximately 78 peer reviews per 100 manuscript submissions, and potential reviewers can more reliably expect to receive higher quality manuscripts to review.
In terms of incentives to review, many publishers offer various types of rewards schemes, such as free journal subscriptions, discount publication offers, and awards or prizes for high levels of reviewing service. Interestingly, feedback on our editorial has suggested that early-career researchers (in particular) are reluctant to undertake peer review obligations because of the time commitment and perceived ‘lack of credit’ and ‘lack of tangible benefit’ that will help them get a job or improve promotion prospects. These comments seem to revolve around the desire for a numerated accounting of effort that they can use as a measure of peer esteem on their CV. Without this, they would rather decline review invitations and spend the time writing their own manuscripts. Regardless of the pros and cons of these perspectives, one way that some journals are providing this kind of numerated accounting is via ‘publons,’ which tracks and validates peer review effort across journals for registered reviewers (and editors). This is something we are currently discussing at Insect Conservation and Diversity.
RW: What would you like to say to readers who may – consciously or unconsciously – be zero-sum reviewers, to try to convince them to adopt another approach?
RD: For researchers out there, nine-tenths of the battle lies in changing their attitude about peer review invitations from ‘burden’ to ‘benefit’. Readers should reflect on the reasons why they are doing science in the first place, and take a moment to consider the genuine competitive advantage they can gain in their career from receiving an advance preview of new developments in the field before they are even published. Peer review provides a low-cost synthesis of how up to date you yourself are with current literature, a benchmark of comparative performance with other researchers in the field, and the opportunity to shape the conceptual and technical direction of your field through critical feedback. So, instead of just complaining about how much ‘rubbish’ is being published these days, and the ‘failure’ of the peer review system to mitigate this, the answer is to be a more constructive part of the solution. There is also great esteem in being invited to be an expert reviewer in your field, and our experience suggests that the benefits accrue in direct proportion to the quantity and diversity of manuscripts that you review. The senior editors at Insect Conservation and Diversity each handle ~60-70 manuscripts per year, and 10-15 manuscripts each for two or three other journals that each of us is an editor on, plus 30-50 peer reviews per annum for other journals in our disciplines. A quick scan of publons peer review ‘heroes’ suggests that some people are doing many times more than this amount as well. These are perhaps aspirational targets for many early career researchers, but why not take the opportunity to have an important influence on the development of your field – think ‘benefit’ not ‘burden’ next time you receive that review invitation.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.