A senior member of a high-profile foreign aid research team has left the project on the heels of a Lancet correction of a heavily criticized paper the team published earlier this month.
Paul Pronyk, who until last week was director of monitoring and evaluation at Columbia University’s Center for Global Health and Economic Development, which runs the Millennium Villages Project, wrote a letter to the Lancet acknowledging errors in the paper, “The effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa: a non-randomised controlled assessment,” originally published May 8. That admission came after Jesse Bump, Michael Clemens, Gabriel Demombynes, and Lawrence Haddad wrote a letter criticizing the work, which was published this week accompanied by corrections to the paper:
Some changes have been made to this Article (see accompanying Correspondence). In the Summary, the last sentence of the Background section should have read “…and compare these changes to local reference data”; the third last sentence of the Methods section should have read “To assess plausibility and attribution, we compared changes to reference data gathered from matched randomly selected comparison sites for the mortality rate of children younger than 5 years of age”; the last sentence of the Findings section has been deleted; and the Interpretation section should have read “An integrated multisector approach for addressing the MDGs can produce rapid declines in child mortality in the first 3 years of a long-term eff ort in rural sub-Saharan Africa”.
In the Introduction, the last sentence should have read “…and compare these changes to local trends”. In the Methods under the heading Procedures, the last sentence of the fourth paragraph has been deleted, and the second sentence of the sixth paragraph should have read “Smears were read by experienced microscopists in local laboratories at baseline and in a research laboratory in Addis Ababa, Ethiopia, in year 3, using best practice techniques”. In the Results, the second sentence of the third paragraph section should have read “The mortality rates in children younger than 5 years of age before the intervention were higher in the Millennium Villages than in the comparison villages (p=0·020; table 1)”; the seventh sentence of the fi fth paragraph has been deleted; and the final paragraph has been deleted.
In the Discussion, the final sentence of the first paragraph has been deleted; and the second sentence of the fourth paragraph should have read “As random site selection across multiple countries was not feasible, we used a pair-matched design to better understand causality and attribution”. The corresponding author has been changed to Prof Jeffery D Sachs, and the Role of the Funding Source statement has been amended to read “PMP had full access to all the data in the study and had final responsibility for the decision to submit for publication”. The appendix of this Article has been corrected. These changes have been made as of May 21, 2012.
Jeffrey Sachs, director of the Millennium Villages Project, tells Retraction Watch that the team’s oversight will be overhauled:
The criticisms from Demombynes and Clemens regarding this timing issue have been on point and helpful. While the progress in the Millennium Villages has been notable, an accurate comparison with local, regional, and national trends in comparable periods is essential. In response to the valuable criticisms, and more generally in order to strengthen the project, I am leading an overhaul of the research organization of the project, including the creation of an independent expert group chaired by Prof. Robert Black (Chair of International Health at Hopkins) to scrutinize, assess, and help to improve the data collection, processing, and analysis. Dr. Paul Pronyk has left the project, and I will co-chair a new faculty research committee with Dr. Cheryl Palm. This new faculty research committee will be the counterpart of the independent expert group.
This is not the first time the group’s work has been questioned. Two of the authors of the Lancet critique also wrote a letter to the American Journal of Clinical Nutrition to criticize a paper published last year in that journal. The issues were similar, but what really caught our eye in the letter was what the authors said was a lack of transparency on the part of the Millennium authors:
[T]he article’s calculations—arrived at through multiple layers of reweighting and matching—cannot be independently checked or replicated by other researchers because the data are strictly internal to the project. Authors of the study informed us that the data will be unavailable to outside researchers until several years in the future, even though results based on those data have already been published in this and other peer-reviewed journals.
The group responded to the criticisms by doing another analysis of their findings, but not issuing a correction. They had this to say about data access:
[T]he project adheres to the requirements of scientific journals in which it is published and the oversight of 11 institutional review boards. We are currently midway through a 10-y evaluation. Survey tools are publically available, indicator definitions are clearly presented, and methods for statistical testing are outlined in detail. Peer reviewers often request additional analyses that may or may not appear in the manuscript or its appendices, and they may request primary data in some instances. Challenges associated with making primary research data widely available have been commented on previously (11). This should be distinguished from secondary analyses of public data sets such as the DHS (supplemental Appendix in the online issue of our article).
Sachs responded promptly to a Retraction Watch request for comment on why the Lancet paper required a correction, while the AJCN paper didn’t. Calling the criticisms “on point and helpful,” Sachs tells us:
Demombynes and Clemens were correct in both cases (AJCN and Lancet) that the outcomes in the Millennium Villages should be compared with national trends during comparable time periods. Using a longer time period for the national data understates the recent national advances. In the case of stunting, sad to say, the recent national advances have not been so fast (the acceleration of national progress has been modest in a number of places). In the case of under-5 mortality, the recent national advances have more generally and fortunately been much faster. I think that AJCN letters and the correction of the Lancet paper are clear in both regards.
Hat tip: Ranit Mishori
It’s difficult to judge solely from this report, but it seems Sachs has manned up and confronted the problems. His lack of weasel-words and clear statement acknowledging the poor analysis, and even better his (unintentional) lack of oversight, are most welcome. Not being a public health scientist, I can’t judge about the data privacy issue, I hope others can.
Not unusual. If you have a large dataset and you plan more mining on it, you keep it for yourself.
Perhaps not unusual, but certainly borderline amoral, as they are receiving donations from large numbers of corporations and charities, including UN organizations which are receiving tax payers’ money through member state contributions. http://www.millenniumvillages.org/uploads/partners/MP-Partners-List-for-Website—15Aug2011.pdf . With a setup like this it could very well be argued that the data has to be made publicly available.
I personally like the approach of large consortia like ENCODE which make their data available for public use pretty much as soon as it has been generated and pre-processed, and ask to not perform large scale data analysis. This allows the community to use the data (and publish results generated from it) in context of narrowly defined research questions, and gives the consortia time to prepare their “big” papers.
My experience with having others use data sets is that they rarely understand what variables are fully rellieable, do not correct/clean data that has not been previously used (and which may have major problems), and do not understand the meaning of a given answer (to the particular respondents involved). The move toward sharing data sets will result in less fruitful analyses. Better to have funding agencies supporting the original researchers for further data analysis. In studies I have worked on, the analyses always ceased because money had run out. I have data sets that would have yielded important information had there been the funds to do additional analyses.
In my experience, cleaning the data is a code word for assuring that the results are what the authors want them to be. My suggestion is that authors make their data available and make sure that process, including the surveys, is so clear that an independently working author can replicate the findings.
I agree entirely with John Kendrick and (obviously) disagree with Sally. It is just sloppy science if the data producers do not document their data properly so others can understand it and either reproduce an analysis or (better) try to get more value out of the data. There is the obvious problem that if only the “insiders” work on the data, their (perhaps unwitting) prejudices — and maybe those of the funding agency?? — will preclude certain lines of analysis.
D G Rosster,
However, if we were to place ourselves in Sally’s shoes, where there isn’t enough time for data analysis, hardly anyone is going to support your efforts to annotate and organize the data for easy consumption by others. I have met plenty of people who were completely willing to share only the raw data as-is, without any attempt to make the others’ analysis easier. I have met only a few handful who are willing to take the time to properly annotate and explain to others what part of the data set means what. It goes without saying that I will keep in touch with people like them for a long time.
Part of the reason experience like Sally’s is all too common is that there isn’t enough management of resources by the principals in charge of the project. When we apply for funding, the time to analyze data is implicitly funded by the grants. The fact that time has run out before the analysis is done indicates poor resource management at all levels of the research team, but mostly from the top.
“Better to have funding agencies supporting the original researchers for further data analysis. In studies I have worked on, the analyses always ceased because money had run out. I have data sets that would have yielded important information had there been the funds to do additional analyses.”
Hence another argument for sharing access to data with the appropriate research community. It’s tempting to keep the data for oneself, but as long as the principals that have collected the data are adequately acknowledged, I really would like to see scientific collaborations flower from shared data.
Another quick quip.
If the collaborators rarely understand what variables are fully reliable, there could be two problems.
One, the collaborators are not adequately trained in your area of research to do the preliminary analysis on their own, at which point you must decide if you’d like to be a part of their analysis as well, after all, it’s your name that’s also included and acknowledged.
Two, if there are good scientific reasons why certain data channels/variables are not reliable, then proper annotation WITH evidence backed explanation should be included in the dataset.
I disagree that move toward sharing datasets will result in less fruitful analysis. Collaborations take effort and time, and they should. Collaborations should not be just one way dumping of datasets. So far in my career, I’ve spent about equal amount of time teaching and communicating with others about the results of my research as I have actually done research.