Did Flint water crisis set kids back in school? Paper saying so is ‘severely flawed,’ say critics

A paper finding kids did worse in school following the Flint water crisis is “severely flawed and unreliable,” according to critics who were deeply involved in exposing the crisis.

The paper has now earned an addendum from the authors, but the critics say it should be retracted.

The authors of the article investigated whether the water crisis in Flint – a period when the drinking water in the Michigan city was contaminated with lead – affected the academic capability of children living there. The authors concluded children in Flint did worse in math after the crisis and more needed special education than before the episode. 

The article was published in Science Advances in March and referenced by major news outlets like ProPublica and The Washington Post.

Soon after, Siddhartha Roy, an assistant professor of environmental science at Rutgers University in New Jersey, and his colleagues sent an email to Holden Thorp, editor in chief of the Science family of journals, expressing concerns about the analysis. In particular, they noted a “confounding” change in the state testing standards at the time of the crisis in Flint that the authors did not account for in their paper. Roy and his colleagues also wrote that the article did not properly acknowledge previous research, and may have lacked the necessary ethical approval. 

The article “reinforces false narratives about the nature of the lead exposure and cause(s) of changing educational performance,” they said in the email seen by Retraction Watch. 

Roy and Marc Edwards, a distinguished professor of environmental engineering at Virginia Tech in Blacksburg, helped expose the water crisis by collecting citywide data with residents that showed elevated lead in water and federal Safe Drinking Water Act violations. 

Edwards and Roy’s group advised the editorial team to launch an investigation based on their concerns. “[I]f found to be legitimate,” they wrote, “we ask that the article be retracted. At a minimum, an expression of concern should be appended immediately while an investigation is underway.” 

In May, Laura Remis, an editor at Science Advances, said the journal had asked the authors to respond to the concerns and requested they submit an addendum to their article with annotated data source and code. 

In the email seen by Retraction Watch, Remis said the authors’ response was sufficient and “we stand behind the conclusions of the paper.” 

In response to claims the paper was unreliable, the authors wrote they “dispute this in the strongest possible terms.”

The study’s analysis of math and reading achievement was “invalid” as it compared different years of Michigan state testing when the standards changed significantly, Roy’s team said. The state’s department of education noted that the tests introduced in 2015 are “very different” and should not be compared to the prior 2014 standard. In response, the authors said their analysis compares only relative performance on tests and does not rely on raw comparisons of different years of data.

In contrast to other peer-reviewed articles, this study also “fails to contextualize actual blood lead levels in Flint children,” Roy’s team said – an omission which ​​creates an impression that the primary adverse health effect from the crisis was lead exposure, which the data in the article doesn’t support. Remis’email said the Science editor did not find merit in this point, so did not ask the authors to respond to this. 

Further, the article “selectively cites and misrepresents evidence on adverse pregnancy outcomes” during this crisis, Roy’s team said. The authors said they believe this concern comes from “a misinterpretation of the text.” 

“While we trust this research project must have required [institutional review board] IRB clearance,” Roy’s team said  “the specific IRB approval code and authorization agency are not mentioned anywhere in the article or supplementary information.” 

In response, the authors said: “The University of Michigan IRB determined that our study used only deidentified data that could be [sic] linked to specific individuals and was therefore not regulated by the IRB. All personally-identifiable information was handled and access [sic] by Michigan Department of Education and the Center for Educational Performance and Information.”

Edwards said they would like to request the raw data from the Michigan Department of Education to redo the analysis themselves, but the cost is prohibitive:

We’ve already spent hundreds of thousands of dollars exposing the [water crisis], and have now years of our lives and hundreds of thousands more, debunking crap science papers about the crisis, so we are doing a cost benefit analysis as to whether it is worth spending another $6000 to possibly correct yet another result that is possibly false. The reality is that because it was already been trumpeted in the media, and many powerful people do not want to see it refuted, even if we exposed the result as crap (and the authors admitted it), it would have little effect to correct the record or perception.

Sam Trejo, an assistant professor of sociology at Princeton University, in New Jersey, and lead author of the paper, told us:

Our authorship team took Roy’s concerns seriously, and we appreciated the opportunity for engagement with our research. Ultimately, we concluded that the points they raised did not threaten any of the paper’s findings. Our team provided Science Advances with a detailed response to Roy’s concerns, and the journal was satisfied with our response. More specifically, the Science Advances editorial staff did not believe it was necessary or appropriate for us to issue a correction or erratum.

This month, the authors issued the addendum – published as an “eLetter” at the very bottom of the article page – explaining their analysis of Michigan’s test scores as well as including a “replication package” to allow others to reproduce their analyses. 

The addendum “is a way of just blowing off our legitimate criticism, allowing the authors to have the last word, without ever giving us the first word, or airing dirty laundry,” Edwards told us. 

“I am surprised that they are calling it an ‘Addendum’ and not an ‘Erratum’ or even a ‘Correction,’” Roy said. “While I am glad the authors were forced to acknowledge that a statewide test change occurred and coincided with the water switch in Flint, they severely downplay its implications.” 

Edwards said he finds the larger context of this paper fascinating:

Research that claims to find something horrible in Flint, seems to get published easily, and is picked up and amplified in the media. Whereas papers that point out problems with those analysis, or shows that the narrative is exaggerated, or might even be harming the people it claims to try and help, has a hard time getting published. The best example of that, is [Roy’s] paper, questioning whether we should continue lying to Flint children, and telling them they were brain damaged by an unprecedented harmful lead exposure event, when all the data shows it never happened. They never stop to think, that the harm of this false story (via a nocebo effect), might overweigh the alleged benefits, for the very people they claim to be helping.

Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].