PLoS ONE expresses concern over flu vaccine paper

via Wikimedia

PLoS ONE has issued an expression of concern over a  2010 paper by Chinese scientists about how the immune system responds to the vaccine against the swine flu.

The article, “Protection Induced on Day 10 Following Administration of the 2009 A/H1N1 Pandemic Influenza Vaccine,” claimed to study 58 subjects given the inoculation (more on that below) and that

Recipients could gain sufficient protection as early as 10 days after vaccine administration. The protection could last at least 60 days. Individuals with a stronger pre-existing seasonal influenza antibody response may have a relatively higher potential for developing a stronger humoral immune response after vaccination with the 2009 A/H1N1 pandemic influenza vaccine.

Some of the researchers were affiliated with the Hualan Biological Bacterin Company, maker of the vaccine they tested and one of several sources of funding for the study — and, it turns out, the reservoir for the patients in the study. (Can anyone say company town?)

According to the expression of concern (which we reproduce here in full):

Following the publication of the article “Immune Protection Induced on Day 10 Following Administration of the 2009 A/H1N1 Pandemic Influenza Vaccine”, the Editors of PLoS ONE noticed that the article described findings from a clinical trial. In line with PLoS ONE’s editorial policies for clinical trials; the article should have included a trial registration number for the study and a copy of the trial protocol.

The Editors followed up on this matter with the authors, who indicated that the trial had not been registered at a public registry. The Editors were also concerned about the documentation and clarifications provided by the authors in response to the request for the study protocol and details on ethical approval for the study. In the light of the concerns raised in relation to the article, the Editors contacted the Institut Pasteur of Shanghai to request an institutional investigation into the research reported in the article.

The Institut Pasteur of Shanghai has now completed their institutional investigation. While the investigation has not questioned the results reported in the article, the enquiry has revealed concerns about some aspects of the study. In the light of the findings of this enquiry, the Editors issue this expression of concern to make readers aware of the following concerns in relation to this trial:

  • The article reports that the clinical component of the study was approved by the Institutional Review Board of Institut Pasteur of Shanghai. However, the investigation committee has stated that the clinical study was in fact approved by the Henan CDC Ethics Committee.
  • The Editors feel that the reporting of the clinical trial in the article is inadequate given that no information is provided regarding the setting for recruitment of the participants. The investigation committee has reported that the participants in the study were employees of Hualan Biological Bacterin Company, China, which is one of the funders of the study, the manufacturer of the vaccine tested, and the employer of several of the authors.
  • The article reports that 58 subjects were recruited for the study, however, the institutional investigation established that approximately 130 volunteers were recruited, while only 58 completed the full protocol.
  • As outlined above, the clinical study was not registered as a clinical trial. PLoS ONE supports the position of the International Committee of Medical Journal Editors (ICMJE) on trial registration and follows the WHO definition of a clinical trial, which constitutes : “…any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes.” In this study participants were prospectively recruited, received a vaccine against the 2009 A/H1N1 influenza virus and their immune response was evaluated following vaccination. We consider the vaccination to be health-related intervention and the measurement of the immune response to constitute a health outcome, and as a result, the Editors consider that the study fulfills the criteria for a clinical trial and should have been prospectively registered at a public repository.

The ethical issues here obviously are quite serious, and we’ve seen papers retracted for similar failings. Indeed, we’ve seen large chunks of a researcher’s published work go down the drain for IRB-related mishaps — take Joachim Boldt, for example.

But the bit about study size strikes us, from a scientific standpoint at least, as potentially far more grave. Assume, for a moment, that the researchers obtained results from 130 patients but cherry picked data from only 58, and it’s immediately evident that their entire analysis — one on which public health decisions very well might rest — is deeply suspect. After all, if the unreported data show that for 72 people (55 percent of the study population) the vaccine isn’t effective until 14 days after administration and lasts only 50 days, Hualan, we have a problem.

We haven’t heard from PLoS ONE about why it felt expressing concern was a better option than retraction.

We asked Damian Pattinson, executive editor of PLoS ONE, why the journal didn’t retract the article.

It was indeed an interesting case and we ended up going to COPE about it, and  it was they that suggested the Expression of Concern option.

For what it’s worth, our main consideration was that the concerns were more to do with potential breaches of our own editorial policies (eg on trial registration) than actual evidence that the work was unreliable. We left open the option of a retraction if the instructional report revealed further issues, but the investigation did not provide clear support for a retraction.

But we did We also reached William Halford, a herpes research at  Southern Illinois University School of Medicine, who edited the flu manuscript for the journal — his first gig for the publication as an academic editor.

Since it wasn’t his area of expertise, he asked the advice of two influenza specalists, both of whom said the paper was worth publishing. However, Halford said he “leaned hard” on the authors to be more specific in many places.

You can tell the difference between a paper in which the authors are trying to be completely transparent and one in which they are deliberately trying to not paint the clearest picture.

If it hadn’t been for the two expert opinions, I probably would have rejected the paper because of the lack of detail. At the end of the day, I remember walking away from the study thinking: It’s my first assignment. I don’t want to be too much of a hard-ass.

We’re sympathetic. Halford had no way of knowing the problems that would befall the paper at the time he was editing the manuscript.

But one person who may be wishing that the journal hadn’t let the article hit the printing press is John Ioannidis, a Stanford researcher who has become well-known for his work on “why most published research findings are false.” Ioannidis was the senior author on a meta-analysis, also published in PLoS ONE, which has so far been the only paper so far to cite the now-corrected work, according to Thomson Scientific’s Web of Knowledge. We’ve asked Ioannidis for comment on whether the correction will affect the meta-analysis, and will update with anything we find out.

Hat tip: Clare Francis

15 thoughts on “PLoS ONE expresses concern over flu vaccine paper”

  1. From its own web page: “PLoS ONE will rigorously peer-review your submissions and publish all papers that are judged to be technically sound.” Shame on the editors, after revealing/judging unsound technical features of this paper, for compounding the mess by invoking an editorial side-step that even Charlie Brown would characterize as wishy-washy,

  2. I don’t think the statement on PLoS ONE is correct in totality. I have experienced their review system. It is not that straightforward at all. They have a long list of editorial board members and handling editors. There are quite a few members about whom I can comment. I might make a list and send it to PLoS ONE to reconsider keeping those members in the editorial board….my concern.

  3. when there’s 13,000 papers a year published (2011) and 26,000 papers due to be published in 2012, what do you expect?

    1. It is like one of those conferences where you pay and become chairperson or key note speaker or invited speaker…include this in your cv for personal benefits…

  4. An imagined future examine question:

    An “expression of concern” is more of a red flag than a retraction? Discuss.

  5. A few weeks back I came across this PLoSOne paper which is another example of a publication which does not fulfil the journal’s own editorial policies:

    It’s just a minor point – missing IRB approval for the animal work in the methods section – but I wonder why they don’t issue check lists to their editors/authors to tick off the requirements for publication. I contacted PLoSOne about this publication but never received a reply…

    1. The point I really wanted to make is this: How on earth is it, that they only notice these issues AFTER publication of a clinical trial? When ever I review a paper which includes human participants/animal experiments, one of the first things I check for is the protocols/IRB approvals… and as this case is described here, I would assume that at least the editor (and the two “influenza experts”), and a minimum of two reviewers should have read this manuscript prior to publication – all with their brains switched off apparently.

      1. some one put up the number of papers published in 2011 is ~13,000. Doing good businesGood going business-wise. Too many editorial board members and academic editors…..but a scary thought – how many of them are …..

  6. Surprise, Plos One rejects manuscripts as well. Thanks to its efficient editorial board!! Quality matters!

  7. As a Chinese researcher, I feel so shameful. I have to say this is only a piece of brick of the whole Great Wall in China.

  8. please have a look at the editorial board member list – find out how they fare in their own research…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.