This is an evolving story, and we will update as we learn more.
A paper in Springer Nature’s Scientific Reports claiming there was essentially no evidence that lockdowns prevented COVID-19 deaths has been retracted.
As of late Monday US Eastern time, while the PDF of the paper was marked “RETRACTED ARTICLE,” a link to the retraction notice’s DOI that had appeared on the page — but did not resolve to anything — had disappeared. The notice appeared at approximately 7 a.m. US Eastern on Tuesday.
Here’s the retraction notice, provided to us by Springer Nature Tuesday morning before it went live:
The Editors have retracted this Article.
Following publication of this Article concerns have been raised about the methodological approach developed by the Authors to evaluate the impact of stay-at-home policies on the reduction of COVID-19-related fatalities. In particular, Meyerowitz-Katz et al.1 show that the approach fails to detect any signal when tested on a synthetic dataset where the ground truth is known, and under specific cases of data subsetting. In addition, Meyerowitz-Katz et al.1 failed to replicate the original results using a synthetic dataset. These suggest that the false negative rate of the approach is prohibitively high to allow for meaningful conclusions to be drawn regarding the impact of stay-at-home policies on COVID-19 fatality rates. The results of Meyerowitz-Katz et al.1 are further confirmed by Góes2 who, using a pure correlation analysis, shows that the coefficients for the impact of stay-at-home policies using the methodological approach developed by the Authors can be zero even with diametrically opposing indices of staying-at-home. Given these concerns, the Editors no longer have confidence that the conclusions presented are adequately supported.
R.S. Savaris, G. Pumi, J. Dalzochio and R. Kunst do not agree with this retraction.
Savaris told us:
This was a terrible decision made by the editors. The basis for the retraction was based on simulated data, they changed the methodology and the reviewers.
In a thread he retweeted Monday evening US East Coast time to announce the retraction, Gideon Meyerowitz-Katz, whose sleuthing efforts on COVID-19 studies have attracted worldwide attention, noted that he and others had published two letters — aka Matters Arising — about problems in the work:
Less than a week after it was published, the paper earned an “editor’s note” — a flag that is not transmitted to typical indexes like PubMed but which Springer Nature has told us in the past is faster than an expression of concern because their policy allows them without a formal response from the authors. It read:
Readers are alerted that the conclusions of this article are subject to criticisms that are being considered by the Editors. A further editorial response will follow once all parties have been given an opportunity to respond in full.
In their letter, Meyerowitz-Katz and colleagues wrote:
That they found no statistically significant differences between regions on these metrics may simply be a function of their chosen methodology and the inherent limitations of the mobility dataset, and might have little to do with the matter at hand. It appears likely that the methodology could not detect an effect of staying at home on mortality even if one were to exist.
As Meyerowitz-Katz notes, the paper “has been accessed nearly 400k times and has one of the highest Altmetric scores of any paper ever.” Many of those who tweeted about the study and cited it in news reports used it to argue against lockdowns.
The paper will be the 202nd to join our list of retracted COVID-19 papers.
Like Retraction Watch? You can make a one-time tax-deductible contribution by PayPal or by Square, or a monthly tax-deductible donation by Paypal to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Regarding the response by Savaris et al.:
> “This was a terrible decision made by the editors. The basis for the retraction was based on simulated data, they changed the methodology and the reviewers.”
As we wrote in our letter, Savaris et al. provided only one – simulated – example as proof that their approach indeed could detect an effect. As shown in our analysis, adding a little bit of noise to the example data eliminated any effect.
We claim that the method is not able to find an effect of reduced mobility on mortality in principle. Savaris et al. could have easily countered all criticism by providing a realistic dataset where their method indeed showed an effect. To the best of my knowledge, they never provided such examples.
I also find the complaint about the change in reviewers interesting. It seems prudent to recruit new, unbiased reviewers if serious concerns about a paper are raised. After all, the criticism not only concerns the published paper but also implicitly the reviewing process that led to publication.
That said, I really appreciate that Savaris et al. openly shared their code and data. If they had not done that, understanding and criticizing their method would have been much harder.