An economics study has been retracted after other researchers identified several inconsistencies in the study’s code and submitted a comment to the journal.
Those critics say the flaws drove the paper’s main findings, but an author of the study says they had no major effect and stands by the results.
The original paper, “Dividend Taxes and the Allocation of Capital,” was published in the American Economic Review in September 2022. It examined the impact of a 2013 increase in the tax rate on dividends in France, concluding an increased rate can encourage the accumulation of capital. The study has been cited four times, according to Clarivate’s Web of Science.
The retraction is only the second in the journal’s history. As Retraction Watch has previously reported, research shows that it is less common for economics papers to be retracted than research in other fields.
The retraction notice published this month states, in part:
The authors are retracting the paper because Figure 4 of the published paper, which pertains to the second result on the increase in investment and which plots the event study difference-in-difference coefficients of the within-firm change in year-to-year investment flows by treated firms, relative to control firms, differs from the figure that was conditionally accepted by the handling editor in two ways.
The first difference, the authors write in the retraction notice, was that “the firm size control variable in the conditionally accepted manuscript was coded in changes but was incorrectly written in the baseline specification as levels.” The authors write that this error was found and corrected during the review and replication process, but they failed to inform the appropriate editor of the changes.
In the second instance, a line of code divided two coefficients by a factor of 1.8. The authors write that while they are not sure how the mistake originated, the data were “stored on remote, external servers that were inaccessible to the authors for prolonged periods of time during the publication process so it is unfortunately not possible to recover past versions to isolate the error’s origin.”
In a response to the retraction, author Adrien Matray, an assistant professor of economics at Princeton University, wrote that both alterations were “made in good faith.” Matray posted the response on his website and linked to it in a tweet. He maintained that neither alteration changed the basic conclusions of the paper.
He also contradicted the text of the retraction notice, which stated that it “explains the authors’ reasons for retracting the published paper.” Matray wrote:
I would like to clarify that it was not my decision to retract the paper. I believe that the scientific concerns could have been addressed in a reply.
He added that given corrections to the study do not change its results, “this paper may not fall within standard publication ethics (COPE) guidelines for retraction.”
Martray did not respond to an email from Retraction Watch.
Charles Boissel, the other author of the paper, left academia in 2018 and was not involved in the alterations that led to the retraction, according to the retraction notice. He was previously a PhD student at HEC Paris.
Erzo Luttmer, the editor of the American Economic Review, told us:
The authors’ retraction notice and the comment published on their article by Bach et al. (https://www.aeaweb.org/articles?id=10.1257/aer.20221432) jointly explain the issues that led to their retraction. The comment explains the nature of the errors and shows how they affect the main results of the article by Boissel and Matray.
In contrast to Matray’s claims that correcting the errors would not change the paper’s conclusions, the comment Luttmer mentioned argues that the alterations essentially invalidate the study’s findings.
The comment was written by Laurent Bach, an associate professor of finance at ESSEC business school; Antoine Bozio, an associate professor at the School for Advanced Studies in the Social Sciences (EHESS); Arthur Guillouzouic, a research economist at the Institut des politiques publiques (IPP) in the Paris School of Economics; and Clément Malgouyres, a researcher at the Paris School of Economics.
In response to an email from Retraction Watch, the authors of the comment wrote that the main conclusions of the paper are “driven by a series of undocumented alterations in the code.”
Referencing Matray’s personal response to the retraction, the authors of the comment said:
One thing that matters for the validity of such an empirical design is that there are no statistically significant coefficients in the pre-treatment period, and both figure 4a and figure 4b in that response clearly present such significant coefficients. More generally, what matters for scientific validity in this context is that coefficients in the post-treatment period do not appear significant when the treatment truly has no effect, and both figures 4a and 4b suffer from this assumption being incorrect due to the inclusion of improper control variables in the analysis, according to our comment and as per [Matray’s] own recognition.
The authors also said that the original version of their comment identified further problems with the study, but that they were asked by the journal’s editor to include only two of these issues.
Update, 1600 UTC, 7/19/23: After this story was published, Adrien Matray responded to our emails, saying they had been lost in his spam filter. In his message, he emphasized that the study had been retracted because of the two alterations to the code discussed in the retraction notice, not because its main conclusions did not hold. He said that a supplemental note on his website addresses “the conceptual flaw in the specification that the authors of the comment present as their most compelling evidence of the lack of results in my paper.”
He also wrote that “there was no evidence that the standard policy for comments, in which external referees are required to weigh in on the initial comment and any response I have, was followed.” He said that when he brought up that he believed the paper did not meet COPE guidelines for retraction, instead of being told that it did meet them, he was told that the journal was not legally bound by these guidelines. Matray said that Luttmer mischaracterized the retraction to the journal’s co-editors as being Matray’s decision, “in order to justify not allowing me to reply to the comment.”
Matray said that the authors of the comment may have a conflict of interest, since one of their working papers examined related reforms and did not find any effects. Their paper has not been published in a journal.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].