The authors of a 2019 Nature paper on hydrology have retracted it after commenters pointed out a slew of errors with the work.
The article, “Global analysis of streamflow response to forest management,” was written by Jaivime Evaristo, of the Copernicus Institute of Sustainable Development at Utrecht University, in The Netherlands, and Jeffrey McDonnell, of the Global Institute for Water Security at the University of Saskatchewan, in Canada. In it, Evaristo and McDonnell produced an estimate of the effects of deforestation on the volume of the world’s rivers.
Their conclusion: “forest removal can lead to increases in streamflow that are around 3.4 times greater than the mean annual runoff of the Amazon River” — nearly enough to double the volume of all the world’s rivers in total.
Disturbing (for those of us not in the field) thought experiment aside, the estimate turns out to be off the mark.
The retraction notice states:
A few weeks after publication of this Article, as a result of comments from James Kirchner and colleagues, we realized that our assembled dataset of paired watershed studies, used to assess the streamflow response to forest removal and planting, contains errors in the percentage change in streamflow associated with land cover modifications. Second, the effects of continent-wide forest removal on streamflow (Table 1 of the Article) are overestimated, because we assumed a starting condition of 100% forest cover. Third, there are serious concerns regarding model validation that need to be assessed using the corrected data. Correcting these honest mistakes goes beyond a simple Author Correction, and therefore we and the Nature editors wish to retract this Article. There are two Matters Arising that accompany this Retraction Note, by James W. Kirchner et al. (https://doi.org/10./s41586-020-1940-6) and by Adriaan J. Teuling & Anne J. Hoek van Dijke (https://doi.org/103810.1038/s41586-020-1941-5). We have decided not to respond because we do not wish to cause further confusion by defending a retracted paper. We are working with Kirchner and colleagues to constructively address the issues raised in a revised paper; if and when it is published we will alert readers by posting a comment to this Retraction.
In their response, Kirchner’s group writes:
Evaristo and McDonnell are valued colleagues of ours, and we greatly appreciate their transparency in making their data and codes available, without which the issues described here would have been much harder to diagnose. We agree with them that streamflow response to forest management is an important issue that deserves a comprehensive analysis, including subsurface catchment characteristics as potential explanatory variables. Readers should also keep in mind that this is not a purely academic exercise. How much, and under what conditions, forests should be cleared is an important policy question with wide-ranging consequences for economies, societies and ecosystems.
We asked Kirchner if, given the flaws, the manuscript should have made it through peer review. His response:
I know the reviewers, and they’re smart people. … It just can be really hard to catch some of these issues without doing a lot more digging than one can realistically expect reviewers to do. The flaws look obvious once you see them, but you *do* need to see them, and there’s some work involved in proving that they are really there. If I were a reviewer I wouldn’t want to say, “I can’t really figure out what’s wrong but it shouldn’t be published because it just doesn’t smell right”, and as an author I wouldn’t want reviewers saying that either.
In this particular case, the system ultimately worked; when the authors saw that there were substantial problems, they decided to retract the paper (and we’re even still on good terms, too).
The retraction was the second for Nature in two weeks.
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Seems like a case where the Reviewers were not scrutinizing the manuscript well enough, does it not?
I’m the guy quoted in the piece above. As that quote says, I know the reviewers of the paper, and they are smart people who did their best.
More generally, I think we really need to avoid holding reviewers responsible for the quality of what gets published. The task of reviewing is thankless enough as it is: there is no compensation, and it takes time from the other work (proposals, papers, teaching, student supervision) you’re supposed to be doing. If we make reviewers responsible for (potentially hidden) flaws in the papers that they review, then we will have made a thankless job into a thankless, all-risk-no-reward job, and nobody will be willing to do reviews anymore.
As a reviewer, one cannot realistically look at every possible angle that might be hiding a potentially fatal flaw. One does the best job that one can, subject to very real constraints of time and energy.
“we really need to avoid holding reviewers responsible for the quality of what gets published”
Then what is the point of peer review? They might as well just mash the “accept” button. After all, there’s no consequences if they let through a bad paper.
Rather interesting situation with a journal that names it’s reviewers. All well unless a major “oops” like this blows up, and then the reviewers are taking the blame in addition to the authors. Seems like an argument for publishing the full reviews but keeping the reviewers’ names out of it.
If the authors publish a bad paper, they get their names on the retraction. If the reviewers OK a bad paper, why shouldn’t they get their names on the retraction?
Because the authors are responsible for the quality of the work that they publish. Reviewers cannot (or at least should not) be held responsible for the quality of papers that they review, because there is no practical way for them to know what unseen problems may be lurking below the surface.
To repeat my comment above: we really need to avoid holding reviewers responsible for the quality of what gets published. The task of reviewing is thankless enough as it is: there is no compensation, and it takes time from the other work (proposals, papers, teaching, student supervision) you’re supposed to be doing. If we make reviewers responsible for (potentially hidden) flaws in the papers that they review, then we will have made a thankless job into a thankless, all-risk-no-reward job, and nobody will be willing to do reviews anymore.
If this were a biomedical paper, one would assume the authors’ were motivated to spruik a cure or similar.
In an article titled, “Global analysis of streamflow response to forest management,”, do you anticipate that the authors’ feel a pressure to spruik the extent of the environmental damage observed in study?
I read biomedical science papers, and often reason that they are overhyped or flawed. Should we start to similarly doubt reports on the environment and/or climate?
“Spruik”? Do you mean “Hype”?
“Spruik” is Australian/British for “hype” (specifically, per Wiktionary and Collins, “To promote a thing or idea to another person”; “to pitch, tout, or hawk”.
Possibly etymologically from Afrikaans/Dutch.