Recently, a biostatistician sent an open letter to editors of 10 major science journals, urging them to pay more attention to common statistical problems with papers. Specifically, Romain-Daniel Gosselin, Founder and CEO of Biotelligences, which trains researchers in biostatistics, counted how many of 10 recent papers in each of the 10 journals contained two common problems: omitting the sample size used in experiments, as well as the tests used as part of the statistical analyses. (Short answer: Too many.) Below, we have reproduced his letter.
Dear Editors and Colleagues,
I write this letter as a biologist and instructor of biostatistics, concerned about the disregard for statistical reporting that is threatening scientific reproducibility. I hereby urge you to spearhead the strict application of existing guidelines on statistical reporting.
As you know, the scientific community is shaken by the unanswered problem of irreproducibility, one aspect of which is the ubiquitous misuse of biostatistics in publications. Statistics serve more purposes than the sole analysis of noisy data: the statistical lexis enables an exchange of knowledge and helps communicate in a standardized manner and careless statistical reporting may endanger reproducibility. In the following, I introduce two of these mistakes that are, alas, shockingly hard to die but that you can help eradicate through simple measures.
The safe-conducts given by the editorial system to articles that do not disclose exact sample sizes are shocking. Science must be based on the possibility to repeat comparable designs, which obviously encompasses the use of similar numbers of observations. Sample sizes given as intervals (e.g. “n=3- 18”), inequalities (e.g. “n>3”) or absurdly nebulous sentences (e.g. “n=4, data representative of 3 rats from 2 independent experiments”) are evident obstructions to reproducibility.
Similarly, it is perplexing to notice the proportion of publications that do not clearly reveal the statistical tests used. A clear attribution of tests must be given, including the post-hoc tests used after analysis of variance. It should not be sufficient to list all statistical procedures in the method section with no indication of which test was used in which figure or table.
The solution is simple: Enforce rigorous policies, policies that often surprisingly exist already in your guidelines for authors. The articles you publish should clearly indicate exact sample sizes that correspond to the number of independent observations as well as the tests used in each analysis. These recommendations are inexpensive and not time or staff-consuming.
I am perfectly aware that editorial policies have improved in the past couple of years, to your credit. However, a superficial skimming through recent articles published in your periodicals on January or February 2017 suffices to notice that the bad habits I have emphasized in this letter have largely outlived all guidelines (see table, below). Such flaws are unworthy of the scientific quality of your journals. Various astute suggestions have been made by others to improve reproducibility. For instance, it is essential to address publication bias, misuse of p-values, use of ridiculously small sample sizes and chronic oversight of corrections for multiple comparisons that inflates the rate of false positives…etc.
However, researchers like you and I know, that at the simple mention of such changes, protests are heard regarding the unbearable costs, in terms of money, workforce or ethics. Every scientist wants sound and reproducible science and is ready to embrace new conducts. But she/he is also justifiably reluctant to be a lonesome reformist that engages more time and money while threatening her/his chances of publication.
Unless we start by the simplest and less expensive issues enacted by collective policies, the scientific community will keep dragging their feet and all these gesticulations will be in vain.
I am mindful that, through this letter, I might eventually incur storms of protests since changes are rarely welcomed with enthusiasm. I am also aware that my non-academic affiliation might bring about skepticism. But scientific reproducibility is an enterprise important enough to outshine individual comfort.
I truly have confidence that we have a shared vision on scientific reproducibility and that you will subscribe to the content of this letter. I sincerely hope that, as editors of highly influential journals, you will take a leading role in this important transition.
The articles included in the table are listed here.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.