Is open peer review the future? The EMBO Journal has offered it since 2009. eLife offers it. They’re not alone, although they’re still in the minority (a fact Irene Hames wishes would change). Elsevier, one of the world’s largest publishers, has tried a pilot of it, too, so we thought it would be worth finding out what happened. We spoke to Bahar Mehmani, Elsevier’s lead for reviewer experience, about the project, and lessons learned.
Retraction Watch (RW): Why did Elsevier decide to launch a pilot of open peer review, and why with these five journals?
Bahar Mehmani (BM): We started in 2012 with Agricultural and Forest Meteorology. The journal’s community and editors wanted to have peer review reports published alongside articles. We started by publishing peer review articles as PDF files in the supplementary documents of the article. However, surveying reviewers and authors as well as looking into collected data showed that peer review reports are not easily found and read. And the community wanted them to have more visibility.
To respond to this need we prepared a workflow enabling the journal to typeset and publish peer review reports with separate DOIs interlinked to the article page (as you found in one example). Despite the manual work involved – and to learn from this practice – we reached out to a small sample of journals asking if they were interested in joining. As a result, these five journals started to publish peer review reports in the format now available on ScienceDirect. We then retrospectively reformatted and published peer review reports for Agricultural and Forest Meteorology going back to 2012.
RW: In 2016, you noted: “The pilot will continue until August 2017, at which point we hope to be ready to offer an open review option to more journal editors. By the end of 2017, we will be able to make it possible for reviewers to choose to publish their review reports in any Elsevier journal.” What is the status of that expansion of the program?
We kept the pilot going until January 2018. At the same time, we designed an automated workflow to collect reviewers’ consent and publish peer review reports (upon the editor’s decision at the publication stage). We are still working on the execution of these workflows.
Once finalized, we will offer the services to all of Elsevier’s journal editors, inviting them and their journal communities to consider publishing peer review reports. At the same time, we are working on providing more stats around the peer review process on article pages on ScienceDirect. For example, as a pilot, we are publishing the handling editor’s name and the number of peer review reports received at the first review round. See examples here and here; they are on a small scale, and we plan to systematically develop and scale them up.
RW: You initially gave reviewers the option of staying anonymous. You reported in 2016 that 45% consented to include their names. Has that percentage changed at all since?
Reviewers can opt in for revealing their identities upon publication of their reports. In 2015, 2% to 45% of the surveyed reviewers (which was not a big sample) said they did opt in. But looking at the full scale and digging into data shows that only 6% in total opted to reveal their identity.
RW: In 2016, 33% of editors said they thought the open reviews were of better quality. Is that motivation for the publisher to expand the program to all journals? Why or why not?
This and our own data analysis comparing reviewer behavior (accepting the review invite and submitting a timely review) before and after introducing this policy for these journal is supportive of the practice. Nevertheless, it is and should be up to community to choose whether to publish peer review reports. As publisher, we are making sure the service is provided to the community should they want to use it.
RW: What additional feedback has Elsevier received from authors and editors about the pilot program and the idea of open reviews?
Authors of pilot journals clearly mentioned in our survey that they support publishing peer review reports alongside their articles. I presented the survey results at the Eighth International Congress on Peer Review and Scientific Publication, and you can find the survey results here.
Also, we should not forget about article readers; traffic on article pages shows that readers are interested in finding out what peer reviewers wrote about the article. I also hear this from researchers wherever I present the pilot.
RW: Are there some fields in which authors and editors are particularly resistant to the idea of open reviews? If so, which ones and why?
Yes. A good example is my own field: theoretical physics. One reason can be that the community doesn’t see the value of transparency in the peer review process, but I am sure once they see the value and at the same time have the incentives for being transparent, they will be supportive. It is hard to convince anyone, least of all researchers, about a concept without providing them with supportive data and analysis. This is exactly what I am working on with a few researchers from PEERE (analyzing reviewer behavior before and after introducing publishing peer review reports to pilot journals). Stay tuned to hear more about it.
Like Retraction Watch? You can make a tax-deductible contribution to support our growth, follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up for an email every time there’s a new post (look for the “follow” button at the lower right part of your screen), or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].