How should scientists think about papers that have undergone what appears to be a cursory peer review? Perhaps the papers were reviewed in a day — or less — or simply green-lighted by an editor, without an outside look. That’s a question Dorothy Bishop, an Oxford University autism researcher, asked herself when she noticed some troubling trends in four autism journals.
Recently, Bishop sparked a firestorm when she wrote several blog posts arguing that these four autism journals had a serious problem. For instance, she found that Johnny Matson, then-editor of Research in Developmental Disabilities and Research in Autism Spectrum Disorders, had an unusually high rate of citing his own research – 55% of his citations are to his own papers, according to Bishop. Matson also published a lot in his own journals – 10% of the papers published in Research in Autism Spectrum Disorders since Matson took over in 2007 have been his. Matson’s prodigious self-citation in Research in Autism Spectrum Disorders was initially pointed out by autism researcher Michelle Dawson, as noted in Bishop’s original post.
Short peer reviews of a day or less were also common. Matson no longer edits the journals, both published by Elsevier.
Bishop noted similar findings at Developmental Neurorehabilitation and Journal of Developmental and Physical Disabilities, where the editors (and Matson) frequently published in each others’ journals, and they often had short peer reviews: The median time for Matson’s papers in Developmental Neurorehabilitation between 2010 and 2014 was a day, and many were accepted the day they were submitted, says Bishop.
Although this behavior may seem suspect, it wasn’t necessarily against the journals’ editorial policies. This is the peer review policy at RIDD:
In order to maintain a rapid rate of review all submitted manuscripts are initially reviewed by the Editor in Chief for completeness and appropriateness to the journal’s stated Aims and Scope. Manuscripts that pass the initial review will be handled by the Editor, sent out to reviewers in the field, sent to an associate editor for handling, or some combination thereof, solely at the discretion of the Editor.
Still, that leaves many, many papers that went through a perhaps inadequate peer review — a phenomenon that is not unique to autism journals. What should be done about such papers? We reached out to several experts to find out their opinion, and present here a selection of the responses. We also want to hear yours – check the poll at the bottom, and feel free to leave a comment.
Let’s start with Bishop herself, who was kind enough to chat with us about whether retractions are in order:
A lot of this work is not very good, a lot of it is about clinical populations, and so we have standing in what looks like a peer reviewed record, things that are low quality or controversial or don’t replicate.
The other concern of a lot of the junior people who have contacted me is the people who have been doing this have been richly rewarded – they get jobs, they get prizes, they leapfrog over the honest people in terms of getting promotions and so on…and that has been greatest theme among people who are more junior, they’re very happy that this behavior will now not get rewarded.
I don’t know whether you would need retractions for that to be achieved…I think I’m reasonably satisfied that these two blogs have created an absolute firestorm, and stopped this from happening. But there is this niggling thing that there are all these papers in the scientific record [that have not been adequately peer reviewed].
Virginia Barbour, chair of the Committee on Publishing Ethics (COPE), sent us a statement from the organization:
We agree that there is a need to put a note on each paper now that questions have been raised over the review of the paper and that the editors are assessing each paper individually. Then the new editors need to do just that. The question of authors’ complicity in the process will have to be part of that review but shouldn’t be a criteria on its own for retraction which should be about more, particularly whether the findings themselves are unreliable.
COPE has issued guidelines for deciding when to retract a paper.
We didn’t survey all the journals Matson edits, but one researcher — Sue Fletcher-Watson, at University of Edinburgh — told us one of her papers also appeared to have an unusually quick review at Springer’s Review Journal of Autism and Developmental Disorders:
I submitted a single-author review paper to RJADD, where Dr Matson is editor-in-chief. I was pleased not to have to respond to reviews – the paper was published in its original, submitted version – but also unsettled by the experience. I don’t think I should have to retract the paper as I submitted it in good faith and wasn’t clear (and I am still not clear) whether the paper was reviewed or not. After all, it is technically possible, if vanishingly unlikely, that reviewers did see the paper but had no substantive recommendations to make. I would be comfortable with an author’s note to the effect that this paper was not externally reviewed however – I’ve no desire to dupe fellow academics or any other readers.
Ferric Fang, a University of Washington microbiologist and Retraction Watch board member who has published papers on scientific misconduct, thinks the issue at the autism journals is the tip of the iceberg:
I see this as part of a broader problem involving the proliferation of scientific journals that do not adhere to accepted practices, which in turn is a form of cargo cult science. It is certainly appropriate to bring attention to journals that fail to engage in a rigorous peer-review process. Articles published in such venues should not be regarded by the scientific community in the same manner as those that have been vetted and revised in response to informed critiques prior to publication. So-called ‘post-publication’ peer review models raise some similar issues: http://scholarlyoa.com/2015/01/06/im-following-a-fringe-science-paper-on-f1000research/.
Retraction is not a suitable tool to address this problem, as this would require cooperation from the same journals that are failing to institute appropriate peer-review. Rather, readers, grant reviewers and institutional promotion committees among others must pay close attention to the journals in which scientists are publishing and ensure that they meet certain standards. This also reveals a problem with simply relying on impact factor and bibliometrics to assess journal and research quality. The case you have cited shows that if a journal permits excessive self-citation and publication without peer review, then that journal and its authors may be able to inflate their numbers in a spurious fashion. Scientists who value their reputations should scrupulously avoid such journals.
Michael Osuch, the publishing director for neuroscience and psychology journals at Elsevier, posted several comments on Dorothy Bishop’s blog spelling out how the two Elsevier journals were changing their policies. A representative confirmed the comments were authentic, but declined to add anything else about what would happen to the papers published under Matson that may have not be adequately reviewed. From the comments:
Under Dr Matson’s editorship of both RIDD and RASD all accepted papers were reviewed, and papers on which Dr Matson was an author were handled by one of the former Associate Editors. Dr Matson and his team of Associate Editors stepped down at the end of 2014, and he retains a honorary position of ‘Founding Editor’, which does not include handling manuscripts.
In the meantime, our publishing staff have been working, alongside a freelance editor, at securing at least two suitable expert referees for papers submitted. We are working to avoid a backlog and limit any potential delays for authors of those papers. Editorial decisions themselves will be made by the new editorial team when appointed shortly. As we move towards making those appointments, we will ensure that quality peer review remains our priority.
In a minority of cases, Dr Matson acted as sole referee. Our focus is now on the future by bring on board new editorial teams whose priority will be to ensure that all accepted articles have a minimum of two referee reports.
Matson himself had few words for us about the papers:
They were all peer reviewed. Look at the post from Elsevier on her blog.
We believe Matson’s quote refers to Osuch’s comment, though he has not responded to a request for clarification. Jeff Sigafoos, one of the editors of the other two autism journals whose articles were often published in Matson’s journal, told the Guardian (in reference to 73 papers published within 2 days of receipt):
The figures you state for 73 papers is routine practice for papers published in RIDD and RASD. A large percentage of all papers published in any given issue of RIDD and RASD appear to have received a rapid rate of review as indicated would happen in the official editorial policy of these journals.
When we contacted Sigafoos, he declined to comment, but said that he felt his comments in the Guardian were taken out of context:
Yes we do have a response that will appear in an appropriate outlet in the near future.
We also made a response to two reporters from the Guardian, but they actually used very little of that and shortened some quotes in a way that I think somewhat undermined the valid points that we were trying to make.
I hope you can appreciate that due to that experience, we would prefer to disseminate our response in a venue that will maintain its integrity.
We’ve reached out to Informa Healthcare, which publishes Developmental Neurorehabilitation; Taylor and Francis, the previous publisher of the journal; and Springer, which publishes Journal of Developmental and Physical Disabilities. We will update if we hear back.
Update 3:40 p.m. EST 4/8/15: The post has been updated to reflect Michelle Dawson’s role.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.