Ben Goldacre has been a busy man. In the last six weeks, the author and medical doctor’s Compare Project has evaluated 67 clinical trials published in the top five medical journals, looking for any “switched outcomes,” meaning the authors didn’t report something they said they would, or included additional outcomes in the published paper, with no explanation for the change. The vast majority – 58 – included such discrepancies. Goldacre talked to us about how journals – New England Journal of Medicine (NEJM), JAMA, The Lancet, BMJ, and Annals of Internal Medicine — have responded to this feedback.
Retraction Watch: When you discover a published trial has switched outcomes, what do you do?
Ben Goldacre: Among the 67 trials we’ve evaluated, in total so far there have been 301 pre-specified outcomes left unreported, and 357 non-prespecified outcomes that were silently added to the reports. This is consistent with the research that’s already been done on the prevalence of outcome switching. We were concerned that this problem has been really well documented, for a very long time, and yet still persists. So we decided to go beyond just publishing anonymous prevalence figures: We’re writing a letter to the journal every time we find a trial that has misreported its outcomes, to correct the record, and to try to elicit progress on this widespread structural problem.
RW: When you contact the journals, what has been their response so far?
BG: The responses from journals have been incredibly variable. People often have ideas about the “character” of different journals, and this may play into that. BMJ has rapidly issued corrections. NEJM has dismissed concerns of outcome switching out of hand. It’s fair to say that so far JAMA have been friendly but ponderous: We are waiting for them to get back to us on their decision about publishing our letters. The Lancet seem to be publishing the correction letters but regarding it as a matter for authors to remedy rather than editors, which we disagree on very strongly, especially as these journals have generally endorsed reporting guidelines like CONSORT. To be clear, we are confident that journal editors in general are strongly committed to improving reporting standards, but there seems to be a rather odd cultural blindspot around policing it. Annals have been the real surprise for everyone: dismissing concerns, writing error-laden “rebuttals”, and even effectively telling trialists that they don’t need to worry about replying to corrections on gross misreporting. There is more to come on that next week.
To be clear, while some of this is undoubtedly disappointing and odd, it’s also useful and interesting. Until we began writing to journals, we only knew that outcome switching was highly prevalent, despite most journals promising to adhere to high reporting standards. Now, from the responses we’ve had, we’re learning why it continues to be so prevalent, we are identifying the recurring misunderstandings and systemic shortcomings. Essentially we’ve solicited qualitative data on the reasons why outcome switching occurs in journals, and it could only have been done by writing these letters.
RW: What’s been the most troubling incident(s) in the journals’ responses to your correspondence?
BG: I think it depends on perspective. NEJM have simply come out and said, effectively: “We don’t care about outcome switching and we don’t care about your letters correcting it”. While we disagree, and we think readers will be surprised to hear that NEJM take that view, it is at least straightforward. The responses from Annals have really surprised everyone, because they’ve been so confused, so internally contradictory, riddled with factual errors, and then they’ve behaved very oddly around publishing responses to their “rebuttals”.
RW: You recently published some correspondence with NEJM editors, such as Deputy Editor Dan Longo, in which he says the editors “view each piece individually and add the data as appropriate based on the judgment of the peer reviewers, the statistical reviewers, and the editors.” In the other correspondence, Longo and other editors say a paper does not contain “any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.” In other words, they may be aware that there is outcome switching, but it’s not impacting the paper enough to warrant a correction or retraction. What is your response to that?
BG: We think this is highly problematic. Of course we don’t think every trialist switching their outcomes is setting out to deliberately mislead readers, or exaggerate their findings. But a culture of permissiveness around outcome switching leads to lower standards, and it gives cover to those who are setting out to mislead. That’s why we have clear guidelines and reporting standards in academic medicine. And to be absolutely clear, there’s nothing wrong with sometimes changing your outcomes, after a trial begins, in the light of new information: But as the CONSORT guidelines say, when you do so, you should discuss and declare this in the trial report. That’s not hard to do, and it prevents readers being misled.
Furthermore, I can’t agree that the changes were trivial. We’re keen to involve students in our projects wherever possible, so we have a team of graduate-entry medical students doing the initial review of all trials in COMPare, then they are reviewed in detail by at least one of our three senior academics (in an incredibly long and very painful meeting, at least twice a week). I asked the coders for their favourite examples of outcome switching from NEJM:
In the ASTRAL-1 trial, one of the outcomes (Incidence of adverse events leading to discontinuation of study drug) was specifically pre-specified as a primary outcome, but reported as a secondary outcome. This is a clear deviation from the original registry entry, and this simple switch significantly changes the meaning of the results, as much less weight is attached to secondary outcomes. This could have been clearly declared in the report, with reasons.
In the trial “Cabozantinib versus Everolimus in Advanced Renal-Cell Carcinoma”, only 4 of 10 prespecified outcomes were reported. Two of the missing outcomes were declared as unreported, however the rest were not. The missing outcomes included “Quality of life anxiety and depression at 6 months”. I think this kind of selective outcome reporting is clinically meaningful, and requires correction of the record.
From Aaron Dale:
One of our letters to NEJM was in response to the ASTRAL- 1  trial, which investigated the effect of the combination of two antiviral agents (Sofosbuvir and Velpatasvir) in patients with chronic hepatitis C virus. The outcomes pre-specified in the registry included several measures of response to antiviral treatments at pre-defined timepoints. When we analysed the paper we found that different time points were reported. Essentially they have switched response at time X to response at time Y and reported these results as effective. Clearly this is an issue because the viral response to treatment may appear more effective at certain time points relative to treatment than others. There can sometimes be good reasons to change outcomes, but these changes should be flagged and justified in the body of the paper to better inform readers.
RW: What are the next steps for the COMPare project?
BG: We’re publishing a paper on our findings so far: To be clear, that’s not just our data on the prevalence of misreporting (because there are already numerous papers showing this problem is extensive) but also the publication rate for our correction letters, and the responses of journals and authors to this misreporting being corrected. We’re going to follow through on all correspondence, from all of our dozens of letters, and all our additional correspondence with journals. Then we have phase 3. I don’t want to sound too cloak and dagger, but we’re not sharing that publicly for now!
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy.