A pair of researchers who have been calling for the retraction of two papers by cardiology researcher Don Poldermans say the New England Journal of Medicine is “not justified” “disappointing” in its refusal to pull the articles.
A little background: Poldermans resigned from Erasmus University in 2011 after having been accused of misconduct. Last week, we reported that the European Heart Journal had issued an expression of concern for a 2001 article on which he was first author.
The New England Journal of Medicine added editor’s notes to two of Poldermans’ articles. But as Larry Husten at Forbes reports, the journal has no intention of retracting the papers:
Readers of the Dutch report [on the Poldermans case] may wonder about the editors’ decision. As I reported this summer, although the Dutch investigators did not find hard evidence of falsified data, they found a number of troubling discrepancies between the trial conduct and the written protocol found in the archive of the Medical Ethics Committee. Poldermans told the investigators that there was an updated version of the protocol but this document has not been found.
In one important discrepancy, the NEJM article states that adverse events were evaluated by an adverse events committee consisting of 2 cardiologists, but both cardiologists told the investigators that they had no awareness that their names were listed in the publication. One said he remembered that Poldermans had asked him to judge some ECGs and the other had no recollection of being involved in the study in any way. Poldermans told the committee that the cardiologists had seen the data for all patients in the study.
Even more troubling are the discrepancies involving the trial’s safety committee. The role of this committee is especially important because the trial was stopped early after the first interim analysis. The NEJM paper states that the decision to stop the trial was made by the safety committee. But the 2 members of this committee told the investigation that they had no recollection of being on the safety committee or of analyzing the study data. One member said that he had given some advice to Poldermans by telephone about “stopping rules.” Poldermans told the investigators that, contrary to the published report, the decision to stop the trial had been made by “the steering committee.”
Two outspoken critics of Poldermans’ work, Darrel Francis and Graham Cole, of the United Kingdom, have been urging the NEJM to retract — without success. They provided us with excerpts from their communications with the journal.
From an August 29, 2014 letter from Francis/Cole to NEJM:
We write to request retraction of the above two articles in the interests of patient safety. We recognize that you may already be considering such action, in which case we hope our attached documents, published today, will be of assistance.
DECREASE I drives meta-analyses which in turn drive even 2014 guidelines. Meta-analysts, senior guideline leaders, political figures, and senior medical journal editors have used non-retraction of the articles as evidence of validity. Since it appears there is no mechanism to answer the questions over DECREASE I, it sits as an RCT roadblock holding back the American and European guideline authorities from grading initiation of perioperative beta blockade as Class III (“Not recommended”), in view of the aggregate of other RCT evidence indicating an increase in mortality.
In an attachment to the letter, which the pair had published in the BMJ, they wrote:
After the 2011 and 2012 Erasmus reports, the ESC press office, European Heart Journal leadership, and 2014 ESC guideline continue to rely on DECREASE I as the only trial to reverse the aggregate result of other trials, which is that initiation of perioperative beta blockade increases mortality. DECREASE I reports an improbably powerful 91% reduction in death and myocardial infarction.
Information missing—When Erasmus University investigated DECREASE I, the patient identities could not be traced for verification of baseline data and outcomes against clinical records.
All members of the committee that assessed events and the committee that stopped the trial, named in the NEJM paper, now deny having had these roles. The say they did not know they had been named in the NEJM. Erasmus could not carry out its planned forensic statistical analysis with the limited information provided by the authors. We suggest that the published papers contain enough meaningful data to decide on the reliability of the reports.
An unusually narrow distribution of heart rate—The range of heart rate of patients at all eight measurement points is so narrow that it is biologically implausible. Each of the eight probabilities is of the order of one in a quadrillion.
Variation between published DECREASE I values and those provided for meta-analysis—Though the values published in NEJM and the values given to meta-analysts might be individually possible, they cannot both be true.
Mathematical uncertainty about the primary endpoint—The odds ratio of the primary endpoint, death and myocardial infarction, is presented with a confidence interval that is mathematically incompatible. Looking more deeply, both seem to be incorrect.
Confusion about which β blocker was used intravenously—The NEJM paper makes contradictory statements regarding whether the intravenous agent was metoprolol or bisoprolol.
Several inconsistencies in the tables—These include apparently fractional patients and a patient with minus 1 myocardial infarctions. The inconsistencies cannot be satisfactorily resolved without the raw data.
But that analysis didn’t sway the NEJM editors, who replied on September 22 with the following:
On receiving your inquiry about the DECREASE I publications, we recognized that the analysis you had performed differed fundamentally from the evaluation carried out by the Erasmus committee. While the Erasmus committee based its evaluation on process assessment – that is, access to data files, review of trial procedures, etc. – you based your evaluation on an assessment of the internal consistency of the trial report. Despite our original decision to base our actions, as is standard practice, on the conclusions of the Erasmus committee, we believed that your evaluation deserved a separate review by the editors.
After careful consideration of the additional evaluation you have carried out, we continue to believe that the available information does not provide clear evidence of scientific misconduct. As a result, it remains our judgment that retraction of the DECREASE I paper published in the New England Journal of Medicine is not justified. Our actions allow people who wish to use the data from this report to make their own judgment as to its veracity.
Thank you very much for bringing your concerns to the attention of the Journal. Following our review of the circumstances, we consider this question closed.
I have to admit that sometimes it just doesn’t make sense to me. Can someone please explain why they decided not to retract the article?! Their explanation seems lacking to me.
My view is that many journals apply the standard of retraction for misconduct beyond *any* doubt. That means that arguments such as “these results are mathematically impossible” or “the methodology stinks” or “the study incorrectly reports …” are not adequate grounds for retraction. To put it bluntly, only a signed confession of misconduct will do.
So that means unless proven by a court of law, or if the person admits… Seems wrong to me. If there is an investigation and there is reasonable doubt, according to my understanding there should be a retraction. You would never get something like that published in the first place.
Yet another clear piece of evidence that there is clearly a problem with the means we use to communicate research and how such communications are evaluated. In this case patient safety might be at stake, but the “system” will not bend.
Chances they would feel the same way if this were a pharma paper?
In itself the NEJM’s statement that the lack of misconduct should inform the decision whether to retract a paper is misleading. The 2009 COPE guidelines (http://publicationethics.org/files/retraction%20guidelines.pdf) are absolutely clear on this. The document states:
‘Journal editors should consider retracting a publication if:
• they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)’
And later on the document states:
‘Retraction is a mechanism for correcting the literature and alerting readers to publications that contain such
seriously flawed or erroneous data that their findings and conclusions cannot be relied upon. Unreliable data may result from honest error or from research misconduct.’
However while the NEJM statement is unhelpful, their hands are pretty much tied by the report from Erasmus. The Committee at Erasmus have not actually said whether the original data are reliable or not – they essentially conclude ‘We have no idea what actually happened’. They seem to blame the discrepancies surrounding the decision to stop the trial on ‘different memories’ and imply that the long period of time elapsed excuses those. To me it looks like a bit of whitewash – but there it is.
Since NEJM have no mandate to conduct their own investigation, with the Erasmus report in their hands one can see why they have refused to retract the paper. To do so they would have to say that the Erasmus report conclusion is itself flawed. Journals have very occasionally retracted papers against Institutional Committee recommendations (Gopal Kundu, JBC is the only example I know). To do that they would have to have absolute, cast-iron evidence that the data are unreliable (e.g. image duplication, or clearly concocted patients e.g. the St. George’s ectopic reimplantation that never took place). The arguments of the UK cardiologists regarding fundamental trial conduct seem persuasive to me but probably don’t quite meet that level of evidence. Clearly the issues are not enough to have forced NEJM’s hand.
In the end you have a kind of conspiracy of inaction here by four parties – the Institution who may well have appointed a ‘soft’ Investigational Committee, the Committee who have failed to rule on what actually happened in scientific terms, the journal who have gone along with their recommendation and the authors who have sat by and essentially said ‘We can’t quite remember what happened but it was all OK’.
‘All that is necessary for the triumph of evil is that good men do nothing’ – and here you have that in spades. The losers are the readers and, quite possibly, patients. The only thing that might help is for a powerful group of respected researchers to write a clear and dispassionate point-by-point critique of the study on purely scientific terms – detailing where and why it breaks fundamental rules of trial conduct (CONSORT) – and hence why it should be removed from meta-analyses – and how the meta-analyses change with its removal.
The editors simply did what is morally and legally right. The authors are not guilty of misconduct unless there is proof that they are guilty. The Erasmus committee concluded that there is no such proof, and the editors did not repeat the investigation.
However, the editor’s note on the paper was very uncommitted. It referred people to the Erasmus web site, but did not give any ‘note of concern’. As ferniglab pointed out, the health of patients are at stake and besides international guidelines are based on this study. This should warrant an editor’s note which should have been much more forthright, stating that serious doubt exists about the presented data.
A follow-up – the Erasmus report makes interesting reading. For example there is a rather farcical section about an independent statistician asked to undertake forensic statistics who ‘concluded that the options for
applying forensic statistical methods to this work were probably limited. This was due to differences between the breaches of academic integrity in Dr Poldermans’ work and those in work by others that Professor Klaassen had previously studied, in which the data was entirely fictitious.’
Translated: The statistician had experience of analysing fictitious data and would be confident of detecting problems if data were concocted. However if there were actually patients, then he / she would not be able to draw any conclusions. Doesn’t exactly sound incisive does it?
Towards the end they do say:
‘…it is remarkable that a number of the analysis files did not distinguish between the non-occurrence of an outcome and the researcher’s lack of knowledge about whether it had occurred. It is evident that this could bias the study conclusions’
Indeed this is remarkable – basically the publications were conflating loss to follow-up with endpoint-free follow-up – a schoolboy flaw. Yet in the end this is all essentially forgiven and they recommend only that:
‘The Committee recommends that the editorial boards of the journals that published the articles
considered here should also be sent this report. On the basis of the information it contains, the
editors can make their own decisions with regard to acting on the Committee’s findings.’
Translated: ‘We don’t really know exactly what happened and so don’t know what to do about it – so you decide – but we suggest doing nothing…’
A damp squib of an ending if ever there was one.
http://www.nybooks.com/articles/archives/2009/jan/15/drug-companies-doctorsa-story-of-corruption/?page=2
“Similar conflicts of interest and biases exist in virtually every field of medicine, particularly those that rely heavily on drugs or devices. It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of TheNew England Journal of Medicine.”
That statement says everything that needs to be said.
The power of the whistle-blower lies not only in the way in which they blow the whistle, but also in the tune that emerges therefrom. Pray, do tell as much as possible, as I suspect, having been an editor of NEJM for as long as that, that important public disclosures are now ripe enough to be told to the public. If things are as bad as you claim, then detailed revelations are urgently required. I guess the same applies to any journal editor that believes that he/she has been witnessing editorial corruption.
PS: By “you”, I mean Marcia Angell.
According to a Dutch TV documentary screened last night, Prof. Polderman was the chairman of the EU committee which made these beta-blockers part of the standard pre-operation treatment in cardiology. Moreover one of his co-authors on the research paper which “proved” their effectiveness was an executive in the pharmaceutical company which manufactured them.
Several experts mentioned that this official policy may have caused hundreds of thousands of unnecessary deaths so far in the EU.