A major medical journal has updated its instructions to authors, now requiring that they publish protocols of clinical trials, along with any changes made along the way.
We learned of this change via the COMPare project, which has been tracking trial protocol changes in major medical journals — and been critical of the Annals of Internal Medicine‘s response to those changes. However, Darren Taichman, the executive deputy editor of the journal, told us the journal’s decision to publish trial protocols was a long time coming:
This change was something we planned prior to COMPARE and were intending to implement with an update of our online journal that is in process. However, the barrier COMPARE encountered in obtaining a protocol for one of the studies in their audit prompted us to implement it earlier.
Previously, the journal reviewed the protocols of submitted clinical trials privately, which made sense at the time, said Taichman:
Careful review of protocols is a standard component of Annals of Internal Medicine’s review process. Knowing that protocol review is a component of Annals review process, most readers are not interested in reviewing the protocols themselves. We also felt that it was more appropriate for investigators to archive the protocols than for multiple copies of the protocol to reside on various journal web sites. However, given that interested readers sometimes encounter barriers despite a reproducible research statement saying that the protocol is available from authors, we will require (for trials submitted after April 19th) that the protocol accompany the published trial report.
Here are the latest instructions for authors regarding trial protocols:
Authors of manuscripts that report clinical trial results must submit the original preenrollment protocol (ideally prepared according to the 2013 SPIRIT standards) with any amendments that were made. All such material must be appropriately dated. For accepted articles reporting clinical trials, Annals will publish the protocol as a supplement to the article.
We also encourage submission of a protocol or an active link to a curated site where the protocol may be found if your manuscript reports a cohort, case-control, cross-sectional or systematic review and meta-analysis study for which there is a protocol. Annals strongly encourages authors of accepted articles reporting cohort, case-control, cross-sectional or systematic review and meta-analysis studies to post the protocol or provide an active link to it.
Taichman added that the journal is also working with other International Committee of Medical Journal Editors member journals to develop a policy to promote data sharing in clinical trials.
Annals hasn’t fared well over the course of the COMPare Project — as Henry Drysdale and Ben Goldacre write on the COMPare blog, the project assessed five trials from the Annals of Internal Medicine, and all five misreported the prespecified outcomes.
When members of the COMPare team alerted the journal to these deviations from trial protocol, it didn’t respond well, according to Ben Goldacre. Earlier this year, he told us:
Annals have been the real surprise for everyone: dismissing concerns, writing error-laden “rebuttals”, and even effectively telling trialists that they don’t need to worry about replying to corrections on gross misreporting…The responses from Annals have really surprised everyone, because they’ve been so confused, so internally contradictory, riddled with factual errors, and then they’ve behaved very oddly around publishing responses to their “rebuttals”.
We asked Taichman to respond to Goldacre’s depiction of the journal’s response. He told us:
As we do with all the comments we receive about published articles, Annals shared COMPare’s comments with the authors to provide them with an opportunity to respond. The comments and any author responses are available on annals.org. We take all concerns raised regarding the studies we publish seriously and evaluate whether an error has occurred. If we conclude that an error has occurred, we publish a correction. We do not, however, always come to the same conclusion as those raising the concern. Readers should evaluate our responses, rather than other individuals’ depictions of them.
The journal’s decision to publish protocols is a move in the right direction, write Drysdale and Goldacre on the COMPare blog:
While this is exactly the kind of result we were hoping for at COMPare – increased transparency to address ongoing structural flaws in science – it is odd that Annals’ editors made this amendment quietly, whilst publicly dismissing COMPare’s methods without evidence to support their claims, defending their current inappropriate and untransparent methods (which they have now reviewed and amended), and resisting constructive positive discussion.
We hope that Annals’ editors will now also move forward and engage in an open and constructive discussion of the misreporting we have identified in all 5 trials assessed in Annals; their dismissal of trial registries; and possible improvements to their processes for identifying and preventing outcome switching in trial reports.
COMPare isn’t the only initiative aimed at improving transparency in clinical trials — in a recent paper on F1000 Research, researchers unveiled a new system to determine if clinical trial protocols have been altered, based on an unexpected system: Bitcoin, the online currency.
More specifically, the system relies on blockchain, the public ledger that underlies the bitcoin system. In the paper, Greg Irving at the University of Cambridge and John Holden of Garswood Surgery converted a trial protocol into a similar ledger-type system using blockchain, which enabled them to quickly check if anything had been changed:
The method described here provides an immutable record of the existence, integrity and ownership of a specific trial protocol. It is a simple and cheap way of allowing a third party to audit and externally validate outcomes and analyses specified a-priori with the findings reported a-posteriori. The method prevents researchers from changing endpoints or analyses after seeing their study results without reporting such changes. Transaction codes could be recorded in scientific papers, reference databases or trial registries to facilitate external verification. Making changes to pre-specified text in a document or trying to bury a protocol in a trial registry would simply not be possible.
There are a few advantages to using such an automated system, they argue:
Firstly, the blockchain would not be confined to the validation of clinical trials. The approach could be used for a whole range of observational and experimental studies where registries do not currently exist. Secondly, the blockchain provides a real-time timestamped record of a protocol. Such precision is important given persistent problems with protocol registration after trial initiation. Thirdly, with over 30,000 trials currently published annually and rising, manual outcome verification is simply not possible.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.