After more than 30 years working with scholarly journals, Irene Hames has some thoughts on how to improve peer review. She even wrote a book about it. As the first recipient of the Publons Sentinel Award, Hames spoke to us about the most pressing issues she believes are facing the peer review system — and what should be done about them.
Retraction Watch: At a recent event held as part of this year’s Peer Review Week, you suggested that journals publish their reviews, along with the final paper. Why?
Irene Hames: I don’t think that saying something is ‘peer reviewed’ can any longer be considered a badge of quality or rigour. The quality of peer review varies enormously, ranging from excellent through poor/inadequate to non-existent. But if reviewers’ reports were routinely published alongside articles – ideally with the authors’ responses and editorial decision correspondence – this would provide not only information on the standards of peer review and editorial handling, but also insight into why the decision to publish has been made, the strengths and weaknesses of the work, whether readers should bear reservations in mind, and so on. As I’ve said before, I can’t understand why this can’t become the norm. I haven’t heard any reasons why it shouldn’t, and I’d love the Retraction Watch audience to make suggestions in the comments here. I’m not advocating that the reviewers’ names should appear – I think that’s a decision that should be left to journals and their communities.
If publishing reviewer reports were to become the norm, the other enormous benefit of this transparency would be that we’d easily and without complicated procedures or checklists expose ‘predatory’/questionable journals. Those with inadequate peer review and poor editorial decision-making would also be revealed. Many editors/journals are finding it harder to recruit reviewers, and there are concerns that this may be affecting quality. At the Peer Review Week panel in Chicago in September, I mentioned the statistic from Publons that the average length of a first-round verified peer-review report in their database was 457 words (median 321). I found this surprising and a bit concerning. Is that enough to do a proper review? Not just to indicate the quality of the work or level of enthusiasm for publication in selective journals, but to provide comprehensive feedback to authors. This is a key feature of high-quality peer review, and not only helps authors improve/rescue their current paper, but also can guide the direction of their research, and so help the research and scholarly efforts of a discipline. This may sound idealistic, but in my 20 years as a managing editor, during which time I oversaw many thousands of manuscripts through peer review and decision, I saw this in action.
RW: How else could the peer review process in academic publishing be improved?
IH: Basically, by making sure systems, processes and policies are fit for purpose so that they help achieve rather than impede efficient and appropriate peer review. This involves reviewing all these regularly, amending them when necessary, and ensuring consistency of information across the journal and manuscript management systems. Very importantly, new issues, developments, and innovations need to be discussed at the editorial board level early on and policies developed. These then need to be communicated broadly, using clear simple language that everyone can understand. If they aren’t, messy situations can arise that cause upset, damage relationships, and introduce delays all round. Preprints and what can be done with peer reviews are recent examples.
There needs to be adequate due diligence, especially when selecting and approaching potential peer reviewers. This shouldn’t need to be said, but the fake peer review cases have thrown up many instances where this hasn’t occurred. I was shocked back in 2012 when the first cases came to light, and now that over 500 retractions due to fake reviews have been reported by Retraction Watch, I’m quite stunned.
Finally, I think there needs to be real thought put into what we can realistically expect individual reviewers to do, and how a comprehensive review can be achieved. Journal articles are nowadays more than just a written paper – they have associated research outputs, e.g. datasets, software, code, protocols, preprints.
RW: You’ve been an advocate for training researchers to peer review articles. Why is that so important, and what might such training look like?
IH: It’s important because being a peer reviewer is an important job – peer reviewers’ assessments and recommendations guide editorial decisions, which in turn influence job appointments, promotions, and grant funding. Peer reviewers also help keep the scholarly literature sound. Up until now there’s been little formal training. That’s changing and many societies and organisations are introducing peer-review training for their communities. This is a good thing, but I worry that some is rather superficial, and also that researchers are in some cases being informed by people in marketing departments with little knowledge of the processes and issues, let alone actual experience.
What should the training look like? The finer details will vary from discipline to discipline, but the basic principles will be similar across all. The training should cover practical and ethical aspects across the whole process: how to respond to peer-review invitations, how to approach doing a review, how/when to communicate with journals during the review process, how to write a helpful and constructive report, how to deal with ethical issues and concerns. But it’s not just reviewers who need training – the same applies to editors. These individuals wield a lot of power, but often come to their roles with little or no experience of managing peer review.
Training in peer review should be part of the wider training in research integrity and publication ethics. Often, problems with research integrity don’t come to light till work is submitted for publication or published – you’ve only got to look at Retraction Watch posts to see the range and scale of problems we’re seeing. Dealing with these is stretching editors, editorial offices, and resources. I know from personal experience that there is considerable lack of knowledge and/or understanding of good research and publication practice among early-career researchers. Incorporating training on this for researchers at an early stage – at postgraduate level – will help minimize the problems editors and journals have to deal with.
RW: Congratulations on being the first recipient of the Publons Sentinel award. (We were honored to be nominated, as well.) How does it feel to have your decades of work towards improving peer review be recognized in this way?
IH: I was deeply honoured to get the award, and actually very surprised when I was told, because of who else was on the shortlist. I’d like to congratulate all the others for the great contributions they’re making to peer-review innovation and accountability, and to keeping standards high. I was, though, really delighted to get the award at this time because next year, after 40 years working in scholarly publishing/communication, I’ll be moving to near-full-time retirement so it’s a very nice way to finish my career.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here. If you have comments or feedback, you can reach us at email@example.com.