In a paper that might be filed under “careful what you wish for,” a group of psychology researchers is warning that the push to replicate more research — the focus of a lot of attention recently — won’t do enough to improve the scientific literature. And in fact, it could actually worsen some problems — namely, the bias towards positive findings.
Here’s more from “The replication paradox: Combining studies can decrease accuracy of effect size estimates,” by Michèle B. Nuijten, Marcel A. L. M. van Assen, Coosje L. S. Veldkamp, and Jelte M. Wicherts, all of Tilburg University:
Replication is often viewed as the demarcation between science and nonscience. However, contrary to the commonly held view, we show that in the current (selective) publication system replications may increase bias in effect size estimates.
During the study, published in the Review of General Psychology, the authors looked at the effect of replication on the bias towards positive findings, taking into account the additional effects of publication bias — the tendency of journals to favor publishing studies that show a statistically significant effect — and the sample size or power of the research.
We analytically show that incorporating the results of published replication studies will in general not lead to less bias in the estimated population effect size. We therefore conclude that mere replication will not solve the problem of overestimation of effect sizes.
In other words, replications, co-author van Assen tells Retraction Watch:
may actually increase bias of effect size estimation. Scientists’ intuitions on the effect of replication on accuracy of effect size estimation are wrong (or weak at best). Replications are beneficial for and improve effect size estimation when these replications have high statistical power.
As long as journals prefer to publish papers that show a positive effect, the authors argue, journals will also prefer to publish replication studies that show a positive effect, regardless of whether it truly exists. Since many scientists believe in the ability of replication studies to reveal the truth about research, some replication studies may actually cloud the truth more than reveal it. Here’s more from the paper:
According to the responses to the questionnaire, most researchers believe that a combination of one large and one small study yield a more accurate estimate than one large study. Again, this intuition is wrong when there is publication bias. Because a small study contains more bias than a large study, the weighted average…of the effect sizes in a large and a small study is more biased than the estimate in a single large study.
All of that, van Assen says, is an argument against positive publication bias, which he calls
currently one of the largest threats to the validity of scientific findings.
Brian Nosek, of the Center for Open Science, has been active in in the push towards reproducibility and replications. He tells us:
The key part of the paper is the answer to the question – what is the culprit?
The answer is publication bias. Replications will increase precision of estimation as long as publication is not contingent on obtaining significant results in original or replication studies. But, if original studies are biased by the publication demand to obtain positive results, then including them in aggregation may not improve the accuracy of estimation.
So, it isn’t really a story about replication. It is a story about how the negative effects of publication bias are hard to undo, even with the presence of some replications.
Much like a bad first date, it is really hard to recover and develop a relationship that you can believe in.
Until the inherent flaws of the publishing system are corrected, scientists should stick to high-powered research, the authors advise:
To solve the problem of overestimated effect sizes, mere replication is not enough. Until there are ways to eliminate publication bias or correct for overestimation because of publication bias, researchers are wise to only incorporate and perform studies with high power, whether they are replications or not.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post. Click here to review our Comments Policy.