For decades, the U.S. FDA has uncovered misconduct in clinical trials but hidden it from the public, according to a new paper in JAMA Internal Medicine.
The study, by New York University journalism professor Charles Seife, looked at 78 publications resulting from trials where the FDA found serious misconduct, including “failure to protect the safety of patients” and data fakery. Only three of those publications mentioned the problems uncovered by the FDA. No retractions or errata were ever issued for any of them.
For example, in one of the three cases:
…data from several patients were excluded from the efficacy analysis because “site monitoring raised questions in regard to certain data at 1 study site.”65(p431) The FDA documents64 allege that none of the individuals enrolled at 1 study site had met the inclusion criteria and that the responsible researcher had fabricated chest radiographs of participants and committed other forms of misconduct.
(Reference 65 reported results of a trial Ivan wrote about several years ago, in coverage that raised some unrelated questions.)
The JAMA Internal Medicine article pulls exactly zero punches:
Conclusions and Relevance When the FDA finds significant departures from good clinical practice, those findings are seldom reflected in the peer-reviewed literature, even when there is evidence of data fabrication or other forms of research misconduct.
Seife also wrote a Slate piece, published concurrently with the paper, which was even more brutal, directly stating that the FDA actively covers up clinical misconduct:
That misconduct happens isn’t shocking. What is: When the FDA finds scientific fraud or misconduct, the agency doesn’t notify the public, the medical establishment, or even the scientific community that the results of a medical experiment are not to be trusted…The FDA has repeatedly hidden evidence of scientific fraud not just from the public, but also from its most trusted scientific advisers, even as they were deciding whether or not a new drug should be allowed on the market. Even a congressional panel investigating a case of fraud regarding a dangerous drug couldn’t get forthright answers. For an agency devoted to protecting the public from bogus medical science, the FDA seems to be spending an awful lot of effort protecting the perpetrators of bogus science from the public.
JAMA Internal Medicine wrote an editorial to accompany the paper (behind a paywall). It noted that the FDA is moving towards more transparency, though their practice of heavily redacting information makes investigative work slow going. Here’s an excerpt:
It is important to note that most FDA inspections are not classified as Official Action Indicated. Seife could not estimate the actual frequency of serious problems owing to the large number of records that were missing or unavailable. And he could not determine how often the violations found by the FDA could reasonably be expected to be acknowledged in an article or how often corrections or retractions were indicated. If the FDA documents had been more accessible and not heavily redacted, however, it is likely that Seife would have linked more published clinical trials to apparent research misconduct.
When the FDA identifies apparent research misconduct, it generally does not make a public announcement or, in the case of a published clinical trial, notify the journal that published the study. It is in the public interest for the agency to make available more information about its compliance and enforcement activities, and it has already begun to do so. In 2009, the FDA announced a transparency initiative; in 2010 and 2011 a task force issued draft proposals, including those to illuminate the agency’s compliance and enforcement activities.
Seife did not contact the journals for reactions, which he normally would have done, he tells us, because this was going in the peer-reviewed literature rather than the popular press.
(Full disclosure: Ivan and Charles are colleagues at NYU, and some of their mutual students helped gather the data for the article). We asked Seife about what it was like working with J-school students, and how the process of digging through the documents went:
One of the wonderful things about having a dozen or so bright students is that you can set them loose on a many-hands-light-work sort of assignment. Go out and find fraud, my pretties! *cackle* So all I had to do was point them in the right direction, and data began trickling in.
The tough part was gathering up all the data and validating it. One advantage we had was that we had no illusions that we’d be comprehensive; the redactions were sometimes way too extensive for us to have hope that we’d get everything. That knowledge kept us from spending too much time beating our heads against the wall trying to crack documents that simply wouldn’t be cracked.
We’ve reached out to the FDA for comment, but for now here’s the one they gave to Reuters. The FDA:
“. . . is committed to increasing the transparency of compliance and enforcement activities with the goal of enhancing the public’s understanding of the FDA’s decisions, promoting the accountability of the FDA, and fostering an understanding among regulated industry about the need for consistently safe and high-quality products.”
We’ve also reached out to COPE for some guidance on the ethics of all this, and will update if we hear back [See update below].
In the Reuters story, Seife agreed with JAMA Internal Medicine that transparency is increasing within the agency. However, he said,
I’m not sure you’ll ever get traction until there’s some incentive behind it.
Update 4:30 p.m. EST 2/11/15: COPE Chair Virginia Barbour got back to us about COPE’s role in this issue:
I had seen this paper and agree it has implications for the reporting of clinical trials.
I am not sure however, this is an issue just for or even primarily for COPE – the responsibility here would seem to me to lie with the authors, the institutions where the trials were undertaken and the funders of the studies – all of whom will presumably have been aware of the FDA findings in relation to the trials they are involved with.
My expectation would be that if such information is available the authors should report it in any article arising from the trial so as to ensure that the information is available to reviewers and editors, and ultimately if it is published and relevant, to readers. Editors should use this information in assessing the papers as should reviewers.
It’s really sad when the best that a government institution can provide about such serious claims is a politically correct statement that actually says nothing, reflects no accountability (x78 times), and shows no immediate desire to address the issue seriously, and transparently. The statement to Reuters sounds more like the marketing banner of a web-site than a frank statement that should have stated, instead: “We are truly concerned about these allegations and will proceed immediately to release all documents publicly that will allow further investigations into these 78 publications to progress. The FDA will do anything and everything in its power to cooperate with the scientific public to ensure that the literature reflects an accurate portrayal of the scientific basis of those papers.” Well, at least that’s what I would say if I were the FDA’s spokesperson…
I am concerned about Dr Seife’s over-reach in concluding in his specific example “the results of a medical experiment are not to be trusted.” It appears the authors, themselves, discovered the malpractice, and pro-actively removed the fraudulent data from their data set. This, of course, would reduce the overall statistical power of their analyses, so if they still saw a statistically significant outcome, it would be even more difficult to obtain, and I don’t see why the published data become necessarily untrustworthy. It also suggests that the authors maintained a reasonable level of scrutiny over the conduct of the trial.
There are two references to JAMA when you mean JAMA Internal Medicine.
Fixed, thanks Marilyn.
From the article: “A clinical site in China taking part in a large trial of apixaban, a novel anticoagulant, had apparently altered patient records. If one were to exclude the data from the patients at that
site, the claim of a statistically significant mortality benefit disappears.179 For this reason, among others, the FDA wrestled with whether it was appropriate to allow the manufacturer to claim a mortality benefit. None of this discussion appears in the literature.”
Apparently, they didn’t wrestle very hard. Here’s the relevant labeling (language is identical in the original and current versions): “ELIQUIS treatment resulted in a significantly lower rate of all-cause death (p = 0.046) than did treatment with warfarin, primarily because of a reduction in cardiovascular death, particularly stroke deaths.”
Note also: the label seems to intend ‘significant’ in the sense of statistical significance, while a plain English reading of it [if you didn’t know what “p=…” meant] is that the reduction in mortality is substantial. Also, that p-value hovers just below the magic 0.05 cutoff. I am mightily reassured.
The FDA’s responsibility is to protect the public health not to correct the scientific literature. All of these incidents can be easily found by searching through the warning letters that FDA sends out to clinical investigators when evidence of non-compliance or fraud is found. All the warning letters are public domain and can be found on the FDA’s website.
If the papers are not corrected, meta analyses will use the incorrect information to provide an overview of the efficacy of a treatment that as a result will be misleading. Thus, if their responsibility is to protect the public health, it _should_ make sure the scientific literature is corrected.
Also, remember the FDA is not a global agency.
If medical literature is to protect public health (if not, what?) then correcting it
is a legitimate of the FDA. the public should not have to be detectives to find what is true and correct. “a falsehood travels round the world while the truth is just getting its boots on”
Bri,
I believe that, according to the article, you are mistaken. To quote a few words from the post,
“When the FDA finds scientific fraud or misconduct, the agency doesn’t notify the public, the medical establishment, or even the scientific community that the results of a medical experiment are not to be trusted…The FDA has repeatedly hidden evidence of scientific fraud not just from the public, but also from its most trusted scientific advisers, even as they were deciding whether or not a new drug should be allowed on the market. Even a congressional panel investigating a case of fraud regarding a dangerous drug couldn’t get forthright answers. For an agency devoted to protecting the public from bogus medical science, the FDA seems to be spending an awful lot of effort protecting the perpetrators of bogus science from the public.”
An excellent article by the outstanding journalist and author Charles Seife.
I am astounded at the near-complete silence from the media on this. I’ve been tracking it with Google News and other than the Slate article, so far I have only found coverage from Reuters, US News, Wired, and WSJ– in terms of mainstream media. A few science pages and other sites I am not familiar with have picked it up. It never ceases to amaze me the stories the media runs with and generates total hysteria over and the stories they completely ignore.
I wouldn’t be surprised to learn that the FDA does not have authority to investigate research misconduct, even in studies under their purview. I suspect that such authority would require an act of Congress, and we know how effective and efficient the U.S. Congress is these days.
Note that I don’t know, and I am not asserting, that the FDA does not have such authority. If you DO know, please let us know.
I think a more appropriate review would have been to look at actions taken by FDA, OHRP and/ORI, and the lack of coordination between them. I don’t know that anyone ever expected “misconduct” to make it into a journal article, but FDA isn’t the only agency looking at this.