Science has issued a permanent expression of concern for a paper reporting a meta-analysis of a database including studies critics have said were “experimentally manipulated.”
The notice, published today, applies to a 2020 meta-analysis measuring population patterns of freshwater and terrestrial insects and predicting what might drive changes in population numbers. According to the notice, the move comes after critics raised concerns about a database, called InsectChange, on which the meta-analysis was based. The database itself was published in 2021 in Ecology, a journal of the Ecological Society of America.
The Science article has been cited 820 times, according to Clarivate’s Web of Science. The Ecology paper has been cited 23 times.
Roel van Klink, a researcher with the German Center for Integrative Biodiversity Research in Leipzig, was the lead and corresponding author for the Ecology paper on InsectChange. He does not agree with the expression of concern and told us in a written statement he did not agree with critics’ assessments. He said the dispute was largely over inclusion criteria – for example, whether it was appropriate to include studies in which researchers manipulated the environments or studies of environments that had been recently changed by invasive species.
“What is also unclear til this day is whether different choices of inclusion criteria would have led to different outcomes” in the meta-analysis, he said. “I would have loved to see a counter analysis based on a different data selection, but that has also not happened during these 5+ years.”
Researchers Laurence Gaume and Marion Desquilbet, along with eight others, critiqued the meta-analysis in a comment published in Science in December 2020, eight months after the publication of the paper. “No attempt was made to weight studies according to their representativeness in terms of geographic location, anthropogenic impact (including farming methods and pesticide use), protected status, or insect assemblages,” they wrote.
Gaume, a researcher at the University of Montpellier in France, told us this week that the submitted comment had “prompted a minimalist erratum” of the Science report, published in October 2020. She said none of the issues they had identified in 2020 was addressed in the erratum, prompting her and Desquilbet, an economics researcher at the Toulouse School of Economics in France, to write a much more extensive critique in Ecology about the post-corrected version. That piece was published in 2024.
The “vast majority of problems we presented require a complete restructuring of the database of the meta-analysis,” Gaume said. She told us the statement she emailed Retraction Watch was cosigned by Desquilbet.
Desquilbet and Gaume wrote in their Ecology critique that “in more than half of the original studies, the factors investigated were experimentally manipulated or were strong — often not natural — disturbances.”
The two pointed to errors in insect count and sampling bias, among other issues. They say they uncovered over 500 mistakes with InsectChange’s methodology and statistical analyses, as BBC’s Science Focus reported last year.
“We argue that the datasets selected by the authors to build their database are not representative of the diversity of insect living conditions around the world and that the database is biased and cannot serve to estimate insect change,” Gaume continued.
Other critics flagged issues with the database in WIREs Water in 2020, noting van Klink and his coauthors “suggested that water quality has been improving, thereby challenging recent reports documenting drastic global declines in freshwater biodiversity.” Those critics argued the results of the meta-analysis “should not be considered indicative of an overall improvement in the condition of freshwater ecosystems.”
Gaume said a retraction of the Science paper was warranted, but said the expression of concern was “better than nothing.” She also said she believed Science should have addressed the issue in 2020, when the first critique was published.
A representative from Science told us the new expression of concern on the paper will be permanent and “has the goal of alerting potential users to ensure they use the most up to date version.” They said they do not plan on looking into any other papers using the InsectChange database.
Gaume expressed unhappiness with the decision. “Marion [Desquilbet] and I are shocked by the way Science is ‘kicking the can down the road’ by essentially referring the problem back to the database, which is supposedly either fixed or in the process of being fixed, while the biased results of the meta-analysis are not being questioned and continue to be cited and influence public opinion, conveying a reassuring message on insects,” she said.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on X or Bluesky, like us on Facebook, follow us on LinkedIn, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].