On Friday, we reported on the retraction and republication of a paper in The Lancet. One of the paper’s authors, Yale’s Harlan Krumholz, took issue with how we characterized the reason for the retraction. We offered him a chance to write a guest post about the situation, which we are pleased to publish below. Please see our editor’s note at the end.
Retraction Watch has grown to play a very important role in promoting responsible conduct of scientific research. Its quest to ‘track retractions as a window into the scientific process’ performs a great service to society. They also have a great responsibility to be accurate in their characterizations of retractions, as all are not alike. I was disappointed that they, in my opinion, mischaracterized the reason for a retraction and republication of one of my papers and did not want to retract their own story (do they have a process to evaluate such concerns?). They said that the retraction occurred because of a major statistical error, when, in my opinion, it was the result of a minor statistical error that affected the results in a very minor way and had no effect on the conclusion. That seems like a more accurate characterization to me. And it makes a difference to the impression of what happened.
Here is the story:
We were part of a recent retraction and republication of an article. We published a paper in Lancet that describe trends in the care and outcomes of patients hospitalized with heart attacks in China. This paper was the first nationally representative study of these patients and highlighted a recent dramatic increase in hospitalizations, the lack of improvement of survival over time, and opportunities to improve the quality of care. The paper has implications for millions of people in China at risk for heart attacks.
After the paper was published online we discovered a very minor weighting error that affected almost all of estimates to a very minor extent. Our study was based on a sampling design that enabled us to evaluation 162 hospitals and make extrapolations to the entire country. This minor error changes most of our estimates very little and did not affect our conclusions to any extent.
We immediately contacted Lancet about the need to make the minor changes. After a thorough review, we proceeded. The paper had not yet been published in paper and so they decided to have the original online paper retracted and republished with the corrections. In my opinion they could have handled it as a correction, but that is their decision. The criteria for retraction in this case would be that the results were unreliable; there was certainly no misconduct. I would argue that the results were reliable and that the minor error required a correction of the estimates. I defer to their judgment – in the end I believe they felt that even though the conclusions did not change they wanted to retract and republish because there were many minor changes.
I took issue with Retraction Watch because they characterized the retraction as occurring because of a major statistical error. Using a weighting scheme that was slightly in error seems to me to be something less than major. I am sensitive about it because I am so proud that our team did the right thing. The statistician recognized the error and rather than ignore something that would not affect the conclusions, she immediately reported it to her supervisor. That person also immediately reported it to the team, who immediately reported it to Lancet and cooperated fully with their inquiry. The estimates were updated and the paper was corrected. The journal then retracted and republished the paper.
I consider this episode to be a good news story. We cannot be perfect, but we should always be honest. I wonder how many teams would have done the same. I just wish Retraction Watch had thought more carefully about their description of the problem.
Editors’ note: We thank Dr. Krumholz for his thorough comment. We agree that the error as he describes it may well be considered minor rather than major. However, as the journal’s actions indicate, the downstream effects of the error were significant enough to warrant retraction — suggesting the choice of words becomes a difference without a distinction.
In the end, everyone agrees — as is clear from the Lancet editorial from which we quoted in the post — Dr. Krumholz and his colleagues acted swiftly and appropriately to correct the mistake and the record. We try always to hold ourselves to that same standard. Therefore, we will remove the word “major” from our post.
Much appreciation to Retraction Watch for being willing to post this blog. It says a lot about their mission and character – and willingness to be open to suggestions. And I am a friend of Retraction Watch – and friends can have disagreements and still support each other. We disagreed on the word ‘major’ – so this blog ensued. But let me say loud and clear – I am grateful for what they are doing – supporting right behaviors and calling out those that… well, could be better. There are few such independent, clear, and principled voices out there today. Just want people to know that the blog is about a particular issue in their characterization of a paper from my team, but that in no way changes my admiration and support for what they are doing.
“We cannot be perfect, but we should always be honest.”
Well, this is a perfect summary if there ever was one. Too bad so many people “featured” on this site lack any honesty or scientific ethics. Thank you for being a professional more concerned about scientific truth than glory. I hope if anything this only adds to your reputation.
Dr. Krumholz is applauded for correcting the literature. I guess the main take-home message -rather than minor vs major – is check the statistical analyses carefully before submission to a journal. It was curious to see Dr. Krumholz’ involvement on the BMJ panel that needed to decide on the retraction of two BMJ papers:
http://www.bmj.com/content/independent-statins-review-panel-report-0
http://journals.bmj.com/site/bmj/statins/Final%20report%20of%20the%20independent%20panel%20310714.pdf
Some interesting praise of Dr. Krumholz’ views on cholesterol:
http://www.drbriffa.com/2012/04/30/prominent-cardiologist-reveals-some-of-the-the-flaws-in-conventional-cholesterol-wisdom/
http://www.drbriffa.com/wp-content/uploads/2012/04/here1.pdf
And some valuable insight about Novartis:
http://www.forbes.com/sites/harlankrumholz/2014/08/30/3-questions-that-remain-after-novartis-blockbuster-trial-result/
Perhaps Dr. Krumholz would care to provide some opinion and perspective of these Novartis-related stories?
http://retractionwatch.com/2014/11/17/retraction-appears-for-faked-study-of-novartis-anti-cancer-compound/
http://www.forbes.com/sites/larryhusten/2014/09/11/japanese-research-scandal-involving-novartis-blood-pressure-drug-widens/
This is a very nice, open statement by the authors.
Nice to see this system more or less working. Looking forward to a future with a rich and open post-review discussion.
Regarding the Novartis questions. Since the publication of their important trial on their new drug, I have call for them to share the clinical trial data. I think that breakthroughs should be scrutinized. Already people are saying that millions will be treated. It would be good to have many others have the chance to analyze the data. Because it is not yet submitted to regulatory agencies that is unlikely to happen. But shouldn’t that be standard operating procedure? How much better would be the conduct of research if everyone knew others would see their work – not just their publications. And the error I reported above would only be detected outside our group by someone who had the raw data. And, by the way, we are working on ways to share that research data too.
I would like to congratulated Dr Krumholz and his co-authors with their nice publication in The Lancet. I am as well delighted that Dr Krumholtz was willing to make this guest post for RW. It is very obvious that you are an emiment scientist and that you are always willing to discuss about all of your findings with anyone. I tend to agree with you that the error was indeed minor. The way how you and your team have solved this problem (and the way how you have reported about it) is great. Great to hear that you work with your team to share the raw research data.
Keeping with the themes of accuracy, transparency and responsive journalism…
The Lancet editorial says “We made this decision because the paper
needed substantive corrections of its findings.”
Dr. Krumholz writes:”This minor error changes most of our estimates very little and did not affect our conclusions to any extent.”
Because it’s been changed without a strike-through, I can’t tell what Retraction Watch, in the original post, said, but I assume it was:
“One of the papers from a massive heart disease study in China, published in the Lancet, has been retracted and republished after the authors noticed a [MAJOR?] statistical error.”
I question whether “findings” and “conclusions” are the same things.
A couple of suggestions for reporting on retractions: don’t use adjectives that aren’t in the retraction notice (ie, major) and use ones in the notice in quotes.
Also, maybe show the actual change in Retraction Watch articles (like with a strike-through) when they are made.
While I too am a fan of Retraction Watch, and have (thus far) never had a paper on which I was an author retracted, I realize that I am not as good a person as Dr. Krumholtz, because I find part of the Retraction Watch response to be entirely self-serving and bureaucrat. Namely, “However, as the journal’s actions indicate, the downstream effects of the error were significant enough to warrant retraction — suggesting the choice of words becomes a difference without a distinction.” Really? Because the journal decided to withdraw the electronic version, so that all the numbers would align in the print version and electronic version, this raises the minor error to a major error? That is, to use the technical term, horse s**t, and appears to be blatant attempt to “apologize without apologizing” for their initial quite erroneous characterization of the retraction.
It is quite possible that Lancet does not correct the published papers. They either retract and then possibly replace or they publish a correction. In this case a correction was probably not suitable due to the number of changes although each change was minor. Some journals do point out that once the electronic version is published that is the final version and they do not make amendments. This is something that journals will have to deal with as electronic publishing becomes the norm. Cochrane just replaces systematic reviews with an updated version when new studies are found, and allows access to the previous versions, which can be slightly confusing.
As to the error, these things happen. I suspect there are huge number of articles out there with minor or even major errors that are left because of the problems of changing. One important paper in diagnostic testing methodology has many errors including transposed tables that must be apparent to many statisticians that have attempted to reproduce the results.