PLoS ONE expresses concern over flu vaccine paper

Following the publication of the article “Immune Protection Induced on Day 10 Following Administration of the 2009 A/H1N1 Pandemic Influenza Vaccine”, the Editors of PLoS ONE noticed that the article described findings from a clinical trial. In line with PLoS ONE’s editorial policies for clinical trials; the article should have included a trial registration number for the study and a copy of the trial protocol.

The Editors followed up on this matter with the authors, who indicated that the trial had not been registered at a public registry. The Editors were also concerned about the documentation and clarifications provided by the authors in response to the request for the study protocol and details on ethical approval for the study. In the light of the concerns raised in relation to the article, the Editors contacted the Institut Pasteur of Shanghai to request an institutional investigation into the research reported in the article.

The Institut Pasteur of Shanghai has now completed their institutional investigation. While the investigation has not questioned the results reported in the article, the enquiry has revealed concerns about some aspects of the study. In the light of the findings of this enquiry, the Editors issue this expression of concern to make readers aware of the following concerns in relation to this trial:

• The article reports that the clinical component of the study was approved by the Institutional Review Board of Institut Pasteur of Shanghai. However, the investigation committee has stated that the clinical study was in fact approved by the Henan CDC Ethics Committee.
• The Editors feel that the reporting of the clinical trial in the article is inadequate given that no information is provided regarding the setting for recruitment of the participants. The investigation committee has reported that the participants in the study were employees of Hualan Biological Bacterin Company, China, which is one of the funders of the study, the manufacturer of the vaccine tested, and the employer of several of the authors.
• The article reports that 58 subjects were recruited for the study, however, the institutional investigation established that approximately 130 volunteers were recruited, while only 58 completed the full protocol.
• As outlined above, the clinical study was not registered as a clinical trial. PLoS ONE supports the position of the International Committee of Medical Journal Editors (ICMJE) on trial registration and follows the WHO definition of a clinical trial, which constitutes : “…any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes.” In this study participants were prospectively recruited, received a vaccine against the 2009 A/H1N1 influenza virus and their immune response was evaluated following vaccination. We consider the vaccination to be health-related intervention and the measurement of the immune response to constitute a health outcome, and as a result, the Editors consider that the study fulfills the criteria for a clinical trial and should have been prospectively registered at a public repository.

The ethical issues here obviously are quite serious, and we’ve seen papers retracted for similar failings. Indeed, we’ve seen large chunks of a researcher’s published work go down the drain for IRB-related mishaps — take Joachim Boldt, for example.

But the bit about study size strikes us, from a scientific standpoint at least, as potentially far more grave. Assume, for a moment, that the researchers obtained results from 130 patients but cherry picked data from only 58, and it’s immediately evident that their entire analysis — one on which public health decisions very well might rest — is deeply suspect. After all, if the unreported data show that for 72 people (55 percent of the study population) the vaccine isn’t effective until 14 days after administration and lasts only 50 days, Hualan, we have a problem.

We haven’t heard from PLoS ONE about why it felt expressing concern was a better option than retraction.

We asked Damian Pattinson, executive editor of PLoS ONE, why the journal didn’t retract the article.

It was indeed an interesting case and we ended up going to COPE about it, and  it was they that suggested the Expression of Concern option.

For what it’s worth, our main consideration was that the concerns were more to do with potential breaches of our own editorial policies (eg on trial registration) than actual evidence that the work was unreliable. We left open the option of a retraction if the instructional report revealed further issues, but the investigation did not provide clear support for a retraction.

But we did We also reached William Halford, a herpes research at  Southern Illinois University School of Medicine, who edited the flu manuscript for the journal — his first gig for the publication as an academic editor.

Since it wasn’t his area of expertise, he asked the advice of two influenza specalists, both of whom said the paper was worth publishing. However, Halford said he “leaned hard” on the authors to be more specific in many places.

You can tell the difference between a paper in which the authors are trying to be completely transparent and one in which they are deliberately trying to not paint the clearest picture.

If it hadn’t been for the two expert opinions, I probably would have rejected the paper because of the lack of detail. At the end of the day, I remember walking away from the study thinking: It’s my first assignment. I don’t want to be too much of a hard-ass.

We’re sympathetic. Halford had no way of knowing the problems that would befall the paper at the time he was editing the manuscript.

But one person who may be wishing that the journal hadn’t let the article hit the printing press is John Ioannidis, a Stanford researcher who has become well-known for his work on “why most published research findings are false.” Ioannidis was the senior author on a meta-analysis, also published in PLoS ONE, which has so far been the only paper so far to cite the now-corrected work, according to Thomson Scientific’s Web of Knowledge. We’ve asked Ioannidis for comment on whether the correction will affect the meta-analysis, and will update with anything we find out.

Hat tip: Clare Francis