Sometimes, a junior member of the team sees things an editor-in-chief misses.
Regular readers know that we’re always delighted when we get a chance to commend researchers and journals for doing the right thing. Here’s an example that sets the standard.
Anesthesia & Analgesia (A&A) is retracting a 2015 paper which purportedly found important differences in patient outcomes based on the quality of their anesthesiologists. The trouble with the article: The authors had made a subtle but significant statistical error that, once corrected, made the results meaningless.
The journal considered the findings of the paper, “The impact of anesthesiologists on coronary artery bypass graft surgery outcomes,” so important that they played it up on the cover of the issue, even running a picture of a King of Hearts. And the paper has been cited 18 times, earning a “highly cited paper” designation from Thomson Scientific’s Web of Science, meaning its citation rate is higher than would be expected. (Five of those citations were in the same issue as the original paper.) According to the abstract:
The rate of death or major complications among patients undergoing coronary artery bypass graft surgery varies markedly across anesthesiologists. These findings suggest that there may be opportunities to improve perioperative management to improve outcomes among high-risk surgical patients.
The handling editor in charge of the manuscript reportedly told the A&A’s (outgoing) editor, Steven Shafer, that the paper was among “the more important papers” he’d managed for the journal.
But even before the paper appeared in print, skepticism of the results arose. As Shafer — a member of the board of directors of our parent non-profit organization — details in an editorial accompanying the retraction notice, an anesthesia resident at Stanford named Lindsay Borg questioned the findings one day when Shafer was reviewing an in press copy with her and other residents. (The paper had been published online, but not yet in print.) As subsequent critics would claim, the results were “a tautology” rather than a real effect.
Willing to eat a foot-long crow sandwich, Shafer describes how he dismissed Borg’s analysis (with what turns out to have been ill-founded statistical authority) — only to find that many other readers were of the same opinion as the doctor-in-training.
Confronted with these concerns, the authors reanalyzed their data, and found to their dismay that their initial conclusions were incorrect. (Shafer writes that he was worried that the replication effort was itself mistaken, so he proceeded with the retraction only after the authors were certain that their new analysis was in fact accurate.)
According to the retraction notice (which, after all this transparency, sits behind a paywall!):
As explained by Glance and Dick, the findings of the 2015 article “The Impact of Anesthesiologists on Coronary Artery Bypass Graft Surgery Outcomes” are incorrect because of flawed statistical analysis. The article is therefore retracted. The data in the retracted article are a subset of a revised analysis appearing in this issue of Anesthesia & Analgesia. The authors’ diligence in identifying the error and correcting the analysis is acknowledged with appreciation.
The new paper, “Feasibility of Report Cards for Measuring Anesthesiologist Quality for Cardiac Surgery,” makes a much more modest claim than the first attempt:
The impact of anesthesiologists on the total variability in cardiac surgical outcomes was probably about one-fourth as large as the surgeons’ contribution. None of the anesthesiologists caring for cardiac surgical patients in New York State over a 3+ year period were identified as performance outliers. The use of a performance metric based on death or major complications for P4P may not be feasible for cardiac anesthesiologists.
Shafer, who has likely done more than any other journal editor to bring order to the chaos of retractions — think Reuben, Boldt and Fujii, for starters — closes his editorial with a few prescriptions.
What can we learn from this? First, the most critical peer review occurs after a paper is published, when it is reviewed by dozens of experts. If something is wrong, it will likely be
identified. Second, peer review is imperfect. Everyone who has received a rejection letter already knows that. Third, science is self-correcting when authors, reviewers, and editors
have the will to do it.
The final lesson is that “retraction” is not a good word to describe the withdrawal of a manuscript when diligent authors discover and report a mistake in their work. There
is too much baggage associated with “retraction.” … “Retraction” seems a harsh reward for honesty. We need a better word to describe the result. We need a term that acknowledges honest error, a type of error familiar to every honest author. We also need to reward honesty with gratitude.
On behalf of the Journal, the Editors, our readers, and the patients whom we serve, I express appreciation to Dr. Glance and his co-authors for their commitment to scientific
To which we say: We couldn’t agree more. From the full explication of the events leading to the retraction, to the publication of several letters, as well as a lengthy response from the authors detailing their process and the error, and even the self-deprecating cover art of a playing card going up in flames, the May 2016 issue of A & A is a textbook example of transparency in scientific publishing. As a likely-not-irrelevant aside, it’s also one of Shafer’s last issues as editor in chief of the journal; his tenure expires later this year. We’re hopeful that the new editor, Jean-Francois Pittet, continues the example his predecessor has set over the years. He certainly has large shoes to fill.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.