Authors retract PLoS Medicine foreign health aid paper that had criticized earlier Lancet study

On May 8, PLoS Medicine published a paper by Rajaie Batniji and Eran Bendavid of Stanford University, about “whether development assistance for health provided to developing country governments increases health expenditures.”

That paper caught the eye of the Center for Global Development’s David Roodman. He began a May 14 blog post about the study, “The Health Aid Fungibility Debate: Don’t Believe Either Side,” as follows:

Have you followed the debate on whether health is “fungible,” i.e., whether giving money to governments to spend on health leads them to cut their own funding for same, thereby effectively siphoning health aid into other uses? It has been like watching the French Open from a center-line seat. Two years ago, a team of authors mostly affiliated with the Institute for Health Metrics and Evaluation (IHME) in Seattle concluded in the Lancet (gated) that health aid has been highly fungible. Now two physician-scholars at Stanford have reanalyzed IHME’s data in PLoS Medicine (quite ungated) and judged the Lancet findings to be spuriously generated by bad and/or extreme data points.

Roodman went on to criticize the paper, criticisms he also noted in a number of comments on the PLoS Medicine site. The authors responded with some clarifications, and the journal noted an error in how the figures were ordered.

Yesterday, the authors retracted the paper altogether:

The authors, Rajaie Batniji and Eran Bendavid, retract their PLoS Medicine essay (1) due to errors in statistical model choice and reporting. Batniji and Bendavid state: “On May 8, 2012, we published an essay that contained a reanalysis of data used by Lu et al. (2) to examine whether development assistance for health leads to displacement of public health spending by recipient governments. In the essay we indicated that we used a fixed effects linear regression, while in Table 4 we reported the results of an analysis using a linear regression without controlling for fixed effects. Upon discovery of this mistake, which resulted from a miscommunication between the authors, we immediately informed PLoS Medicine and moved to correct the record. Our essay concluded that the findings in the analysis by Lu et al. were strongly influenced by outliers and questionable data. However, when our analysis is done using a linear regression with fixed effects, the results show a similar effect size and significance level to that indicated in the original article by Lu et al. As a result of this error, we now believe the following statements in our essay lack justification:

1. “the association between DAH and displacement of government health expenditures is not robust after exclusion of a small subset of data.”

2. “The trends [in the analysis by Lu et al.] are driven by outliers…”

3. “While there does appear to be an association, it is too tenuous, too dependent on problematic model selection, and inconsistent.””

The PLoS Medicine editors agree with the need for the Retraction and accept the authors’ explanation of their error. During the editorial handling of the essay, the paper was reviewed twice by our statistician: once following peer review by content experts and revision by the authors, and again following a second author revision and before acceptance. We apologize to readers for any inconvenience caused by the publication of this article in the journal. We have invited short perspectives from Batniji and Bendavid and from Lu et al. to expand upon evidence for aid displacement in health.

References:
1. Batniji R, Bendavid E (2012) Does Development Assistance for Health Really Displace Government Health Spending? Reassessing the Evidence. PLoS Med 9(5): e1001214. doi:10.1371/journal.pmed.1001214

2. Lu C, Schneider MT, Gubbins P, Leach-Kemon K, Jamison D, Murray CJL (2010) Public Financing of Health in Developing Countries: A Cross-National Systematic Analysis. Lancet 375: 1375-1387.

The error the authors describe fits with what Roodman wrote in his blog post:

Their preferred method is described in just one sentence, which contains a phrasing, ‘country fixed effects, clustered by country,’ that I have never seen before, and I think reveals confusion. It appears to me that the estimator does not include fixed effects, but merely clusters standard errors by country.

Roodman told Retraction Watch that really, there was “more wrong with the paper than this.”

For example, the idea I start with in the blog post—that they weren’t interpreting their own results correctly—is distinct from the methodological problem they concede.

Still, he praised the authors for swift action.

But the bottom line is that they’ve retracted their bottom line, as I believe they should.

And:

I think the authors have behaved well since publication. They shared data and code (though PLoS’s policies requiring data sharing and encouraging code sharing may also have played a role), and they seem to have moved pretty quickly to retract.

Roodman notes that the retraction does not end the French Open-like debate. Speaking of the original Lancet paper, he says:

This doesn’t mean that Lu et al. are vindicated, only that this criticism doesn’t work.

Roodman’s colleague Michael Clemens, we should note, was recently involved in the correction of a Lancet paper, also on foreign aid.