Paul Pronyk, who until last week was director of monitoring and evaluation at Columbia University’s Center for Global Health and Economic Development, which runs the Millennium Villages Project, wrote a letter to the Lancet acknowledging errors in the paper, “The effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa: a non-randomised controlled assessment,” originally published May 8. That admission came after Jesse Bump, Michael Clemens, Gabriel Demombynes, and Lawrence Haddad wrote a letter criticizing the work, which was published this week accompanied by corrections to the paper:
Some changes have been made to this Article (see accompanying Correspondence). In the Summary, the last sentence of the Background section should have read “…and compare these changes to local reference data”; the third last sentence of the Methods section should have read “To assess plausibility and attribution, we compared changes to reference data gathered from matched randomly selected comparison sites for the mortality rate of children younger than 5 years of age”; the last sentence of the Findings section has been deleted; and the Interpretation section should have read “An integrated multisector approach for addressing the MDGs can produce rapid declines in child mortality in the first 3 years of a long-term eff ort in rural sub-Saharan Africa”.
In the Introduction, the last sentence should have read “…and compare these changes to local trends”. In the Methods under the heading Procedures, the last sentence of the fourth paragraph has been deleted, and the second sentence of the sixth paragraph should have read “Smears were read by experienced microscopists in local laboratories at baseline and in a research laboratory in Addis Ababa, Ethiopia, in year 3, using best practice techniques”. In the Results, the second sentence of the third paragraph section should have read “The mortality rates in children younger than 5 years of age before the intervention were higher in the Millennium Villages than in the comparison villages (p=0·020; table 1)”; the seventh sentence of the fi fth paragraph has been deleted; and the final paragraph has been deleted.
In the Discussion, the final sentence of the first paragraph has been deleted; and the second sentence of the fourth paragraph should have read “As random site selection across multiple countries was not feasible, we used a pair-matched design to better understand causality and attribution”. The corresponding author has been changed to Prof Jeffery D Sachs, and the Role of the Funding Source statement has been amended to read “PMP had full access to all the data in the study and had final responsibility for the decision to submit for publication”. The appendix of this Article has been corrected. These changes have been made as of May 21, 2012.
Jeffrey Sachs, director of the Millennium Villages Project, tells Retraction Watch that the team’s oversight will be overhauled:
The criticisms from Demombynes and Clemens regarding this timing issue have been on point and helpful. While the progress in the Millennium Villages has been notable, an accurate comparison with local, regional, and national trends in comparable periods is essential. In response to the valuable criticisms, and more generally in order to strengthen the project, I am leading an overhaul of the research organization of the project, including the creation of an independent expert group chaired by Prof. Robert Black (Chair of International Health at Hopkins) to scrutinize, assess, and help to improve the data collection, processing, and analysis. Dr. Paul Pronyk has left the project, and I will co-chair a new faculty research committee with Dr. Cheryl Palm. This new faculty research committee will be the counterpart of the independent expert group.
This is not the first time the group’s work has been questioned. Two of the authors of the Lancet critique also wrote a letter to the American Journal of Clinical Nutrition to criticize a paper published last year in that journal. The issues were similar, but what really caught our eye in the letter was what the authors said was a lack of transparency on the part of the Millennium authors:
[T]he article’s calculations—arrived at through multiple layers of reweighting and matching—cannot be independently checked or replicated by other researchers because the data are strictly internal to the project. Authors of the study informed us that the data will be unavailable to outside researchers until several years in the future, even though results based on those data have already been published in this and other peer-reviewed journals.
The group responded to the criticisms by doing another analysis of their findings, but not issuing a correction. They had this to say about data access:
[T]he project adheres to the requirements of scientific journals in which it is published and the oversight of 11 institutional review boards. We are currently midway through a 10-y evaluation. Survey tools are publically available, indicator definitions are clearly presented, and methods for statistical testing are outlined in detail. Peer reviewers often request additional analyses that may or may not appear in the manuscript or its appendices, and they may request primary data in some instances. Challenges associated with making primary research data widely available have been commented on previously (11). This should be distinguished from secondary analyses of public data sets such as the DHS (supplemental Appendix in the online issue of our article).
Sachs responded promptly to a Retraction Watch request for comment on why the Lancet paper required a correction, while the AJCN paper didn’t. Calling the criticisms “on point and helpful,” Sachs tells us:
Demombynes and Clemens were correct in both cases (AJCN and Lancet) that the outcomes in the Millennium Villages should be compared with national trends during comparable time periods. Using a longer time period for the national data understates the recent national advances. In the case of stunting, sad to say, the recent national advances have not been so fast (the acceleration of national progress has been modest in a number of places). In the case of under-5 mortality, the recent national advances have more generally and fortunately been much faster. I think that AJCN letters and the correction of the Lancet paper are clear in both regards.
Hat tip: Ranit Mishori