If audits work for the Internal Revenue Service, could they also work for science? We’re pleased to present a guest post from Viraj Mane, a life sciences commercialization manager in Toronto, and Amy Lossie at the National Institutes of Health, who have a unique proposal for how to improve the quality of papers: Random audits of manuscripts.
Skim articles, books, documentaries, or movies about Steve Jobs and you’ll see that ruthlessness is the sine qua non of some of our greatest business leaders. It would be naïve to assume that scientists somehow resist these universal impulses toward leadership, competition, and recognition. In the white-hot field of stem cell therapy, where promising discoveries attract millions of dollars, egregious lapses in judgment and honesty have been uncovered in Japan, Germany, and South Korea. The nature of the offenses ranged from fraudulent (plagiarism and duplication of figures) to horrifying (female subordinates coerced into donating their eggs).
When a researcher embraces deception, the consequences extend well beyond the involved parties. Former physician Andrew Wakefield published a linkage between MMR vaccines and autism with overtly substandard statistical and experimental methods, while hiding how his financial compensation was tied to the very hysteria he helped unleash.
Let’s ask some hard questions. Should it have taken 12 years for the Lancet to retract Wakefield’s article? Do we exercise sufficient caution and skepticism when an author claims to have derived the newest miracle stem cell? Should we assume professional scientists have an adequate understanding of research ethics, experimental design, and transparency?
These questions don’t just apply to instances of blatant misconduct, but also to the ongoing, highly visible problem of reproducibility in science. Most often, reproducibility concerns stem from issues with statistical and data analyses. What if a study’s reproducibility and compliance with established guidelines were assessed comprehensively, before it was published?
When it comes to enforcing compliance, there is an established method that any taxpayer or corporate accountant has a healthy fear of: the audit. We propose a systematic and independent audit of research manuscripts before they are reviewed by a journal’s panel of referees and editors.
Here we outline an approach that draws on the arms of the Internal Revenue Service (IRS) and corporate auditing methods, adapting the concept for the unique needs of scientific research.
- The IRS does not audit every taxpayer (thank goodness!); in fact only around 1% of taxpayers faced an audit last year. However, this low risk of audit still keeps tax code violations down to around 17% of filings. We propose the formation of an examining body mandated to audit a small percentage of submitted scientific manuscripts. Unlike tax audits that are triggered by predictable taxpayer filing behaviors, scientific audits would be randomized so that all authors perceive an equal risk of being examined.
- To maintain impartiality, the examining body must be independent of author-affiliated institutions (universities, hospitals, etc.) and journal editorial boards.
- The examiners should be empowered to request raw data from the authors, perform their own verification of the authors’ data analysis and conclusions, and send an audit report to the editors of the intended academic journal.
- One possible embodiment of the examining body is a collective of consulting specialists including statisticians, professional graphic artists, audio engineers, etc. This group would offer multidisciplinary, ad hoc expertise that cannot realistically be replicated by a journal’s editorial board.
- To reinforce the presence and value of these audits, the published article should include a brief acknowledgement that the manuscript was independently reviewed by the examining body, along with a link to the unabridged audit report.
The Independent Audit System has several benefits. For one, it provides an unbiased verification that experiments are conducted ethically, results are calculated appropriately, and conclusions are based on the dataset rather than potential newsworthiness. It also prevents bad papers from getting out — a pre-publication audit of Andrew Wakefield’s discredited study could have revealed statistical irregularities, conflicts of interest and methodological flaws, and prevented serious harms to public health.
An audit system also yields opportunities for specialists in image manipulation, statistics, big data, etc. to participate in scientific review on an as-needed basis. This group would be a logical home for the domain expertise of postdoctoral fellows, many of whom have already participated in traditional peer review. We envision a flexible, lean organization, unencumbered by traditional limitations on geography or work schedules. The group, by definition, would draw on diverse skill sets. Finally, an audit could ultimately create an opportunity for authors to choose to be audited as an expression of confidence in their experimental design and conclusions, in the same way that architects and building managers willingly undergo audits for LEED certification.
Our proposed auditing body also has the advantage of providing an independent third party review, rather than imposing extra commitments onto the editorial boards of participating journals, oversight committees of universities, or overworked reviewers.
There’s another big question: Who pays for this audit? Funding could take many forms – major organizations for scientific inquiry such as the National Academy of Sciences, the American Association for the Advancement of Science, the Gates Foundation, the National Institutes of Health, the National Science Foundation and the Office of Research Integrity make logical partners, and could stipulate that their grant recipients would be subject to audit. Employers of the actual auditors can also contribute to this endeavor by simply maintaining salary coverage during the brief periods that their employees are contributing to scientific review. Lastly, small fees could be obtained from the authors under review, as well as their intended journal. These fees would be offset by the provision of a novel deliverable: a professional scientific audit report.
We are not the first to propose enhanced scrutiny of scientific publications; the Wakefield scandal ultimately led to calls for “an external regulator overseeing research integrity,” although the suggested placement of that position within each university strikes us as a potential conflict of interest. While there is no guarantee that our team of auditors would catch the issues missed by traditional peer review, the opportunity to deliver objective, multidisciplinary, critical review is an important step forward in scientific transparency. Unlike traditional peer reviewers who may prioritize this responsibility well below their own endeavors (or delegate it to a graduate student), each ad hoc auditing panel would be convened for a specific manuscript and can therefore operate within a reasonable window of time to minimize the delay experienced by the authors. A final issue is access: how does the auditing body find and obtain pre-publication manuscripts in a fair and transparent manner? We propose a random number generator that selects a journal, a day, and a time for inquiry. The selected journal board is then asked to submit the corresponding non-reviewed manuscript to the audit body. This avoids the need to access a mega-repository of manuscripts from multiple journals. If journals choose not to comply, that can be publicized as well, perhaps here at Retraction Watch.
The auditing body we propose is designed to elevate the standards for gaining, analyzing, and disseminating information. As stated expertly by Begley and Ioannidis, “It is not at all clear how reasonable recommendations will be implemented or monitored while they remain voluntary.” By avoiding change we simply enable future scientific misconduct, which leads the public to cast aspersions far beyond the actual perpetrators. If we wish to build a better relationship with the public, then increased transparency is non-negotiable. Sunlight is said to be the best disinfectant.
Viraj Mane, PhD, provides project management, fundraising, and commercialization services to scientists in Toronto, and is also a designer and purveyor of absurd clothing. Amy Lossie, PhD, is a Health Science Administrator at NIH and the co-founder of the Beautiful You MRKH Foundation, Inc.
Update 2/6/16 4:34 p.m. eastern: Update: The authors would like to acknowledge the previous work in this area by Adil E. Shamoo; the concept of random auditing has also been raised in the past by Drummond Rennie, deputy editor (West) at JAMA.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy.