When economist Jason Hockenberry looked at data comparing some of the financial issues facing different U.S. hospitals, he was surprised by what he saw.
Hockenberry was examining the effects of a recently introduced U.S. program that penalizes hospitals with relatively high rates of readmissions for certain conditions by reducing Medicare payments. Although Hockenberry expected hospitals that serve low-income and uninsured patients to have more readmissions (and therefore more penalties), he saw these so-called “safety-net hospitals” had been steadily improving their performance since the program began in 2012, and had faced fewer penalties over time.
The results were so striking, they ended up in JAMA on April 18, 2017. But within one week after publication, Hockenberry learned outside researchers had raised questions about the analysis.
The outside researchers thought the authors had incorrectly categorized some of the safety-net hospitals. After looking into their concerns, Hockenberry — based at Emory University in Atlanta — realized the analysis did contain errors that affect the findings. This week, he and his co-authors retracted the article, replacing it with a corrected version. The new paper still reports that the gap between the penalties faced by safety-net and non-safety-net hospitals is closing — but not for the reasons they initially thought.
Hockenberry told us:
The error was an honest mistake. I wish it hadn’t happened, but the correct information needs to be out there.
The paper explored the extent to which two government programs—Medicare’s Hospital Readmissions Reduction Program (HRRP) and the Hospital Value-Based Purchasing program (HVBP)—put safety-net hospitals at a financial disadvantage by reducing payments for excess readmissions. But within a week after the paper was published, researchers who analyzed the same publically available data contacted the journal with their concerns.
Hockenberry immediately got to work checking the data, and quickly discovered the researchers were right: The authors had made a coding error, which put some safety-net hospitals in the “perfect performance” category (0), when they should have been classified as “expected performance” (1):
That same afternoon, I replied to the editors and told them about the coding error. As gut wrenching as the discovery was, we wanted to correct the literature as quickly as possible.
Hockenberry said the authors and editors discussed the best path forward:
Given that the error changed the implications of the paper, we decided the retract and replace was the best approach.
A retract and replace has become an increasingly popular option for JAMA and a few other journals, to allow authors to correct honest but pervasive mistakes.
The authors got to work revising the paper, “incorporating the different but correct information.” Hockenberry said that the “main thrust of the paper was largely unaffected”—the gap in financial penalties between safety-net hospital and other hospitals had indeed narrowed over the past few years. But the reason for the narrowing had changed: They found that non-safety-net hospitals are being penalized more than previously.
The authors provided a detailed explanation of the error in the retraction and replacement notice, published on September 19:
Readers noticed that the excess readmission ratios were lower than would be expected. In reviewing our data, we found that we had made a coding error while conducting initial sensitivity analyses. We assigned hospitals that had an insufficient volume of patients for the given HRRP conditions as having an excess readmission ratio value of 0, rather than a value of 1. Assigning a value of 1 implies that the low-volume hospitals’ readmissions were no different than expected, which is what we intended. In contrast, assigning them a 0 implied perfect performance (no readmissions), which we did not intend and which overstates performance. In addition, in reviewing our analytic files, we discovered we had applied exclusion criteria to our sample of hospitals that were salient to earlier published work but not to this study, resulting in 163 hospitals being excluded from the analysis.
The paper “Comparative Trends in Payment Adjustments Between Safety-Net and Other Hospitals Since the Introduction of the Hospital Readmission Reduction Program and Value-Based Purchasing,” has not yet been cited, according to Clarivate Analytics’ Web of Science. (The original, retracted version of the paper is now part of the supplementary data.)
The authors goes on to explain the extent to which the errors affected their results:
Our original conclusion that the performance of safety-net hospitals on the HRRP improved between FY 2013 and FY 2016 is unchanged. However, that the improvements may have been associated with improvements in readmission rates for pneumonia and heart failure is no longer correct. Rather, as we now point out in the replacement article, “The performance of safety-net hospitals on the HRRP has improved between FY 2013 and FY 2016 as reflected in the penalties. This does not appear to be driven by meaningful reductions in the excess readmission rates at safety-net hospitals but rather may have been driven by the design of the program and addition of arthroplasty readmission to the program in FY 2015, with mean excess readmission ratios above 1 for both types of hospitals in 2015 and 2016.”
Hockenberry also noted, he is “very thankful to the team who saw our article” and identified the error, given the potential policy implications of the work:
If safety-net hospitals are unfairly exposed to higher penalties, it may thwart their safety net mission.
Hockenberry explained:
The broader issues are the extent to which these penalties, even though they are now similar in both types of hospitals, will have a disproportionate impact on the populations served by safety net hospitals and whether as the program expands the number of conditions nearly all hospitals will essentially start incurring some penalty.
Update, 14:45 UTC, September 22 2017: We’ve heard from Annette Flanagin, Executive Managing Editor and Vice President, Editorial Operations, JAMA and The JAMA Network:
This was an honest coding error that does not change the author’s overall conclusion that the performance of US safety-net hospitals in the Hospital Readmission Reduction Program improved between 2013 and 2016. However, there were errors and now corrections in some of the specific findings as detailed by the authors in the Notice.
The decision to accept this Research Letter was not based on the specific findings per se but rather on the merits of the study design and importance of the topic. The editors believe that the study is still valid and with the authors published the Notice of Retraction and Replacement to correct the original article.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.