PLOS ONE has issued an expression of concern after the authors of a controversial study about chronic fatigue syndrome declined to share some of their data.
In an unusual move, the journal included the authors’ response to the expression of concern (EOC), in which they strongly argue against the notice, and “do not accept that it is justified.”
In 2015, following public requests to review the data, the journal issued an “editor’s note” on the paper, noting the journal’s policy that authors make data and materials available.
There have been numerous requests for data from the “PACE” trial, as the clinical trial is known. Patients and advocates have long disputed the results, arguing that suggesting cognitive behavior and graded exercise therapy could cause harm.
In the latest notice, the journal says it consulted two editorial board members about the paper, a 2012 sub analysis of a controversial clinical trial on chronic fatigue syndrome. The journal then asked the authors to provide the data behind five tables, which would enable researchers to replicate the cost-effectiveness analyses the authors report for different therapies — including graded exercise therapy, which some patient advocates believe could be harmful.
As with previous requests for data, the authors refused to provide it, citing patient confidentiality and consent. The notice explains:
In consideration of the requirements for ethical oversight of data access that may apply to datasets involving human subjects, we contacted the authors and Queen Mary University of London, where the dataset is held, to request that steps be taken to develop a mechanism that would allow requests for data to be independently reviewed and the data released in accordance with our policy while respecting patient privacy. The authors and Queen Mary University of London shared the data policy in place at the institution, however we consider that aspects of the existing framework impose limitations and conditions not aligned with our editorial policy.
The authors proposed a compromise, according to the journal:
The authors have offered to release aggregated data from the study but have reiterated reservations about the public release of individual-level patient data. The journal policy does not require public release of anonymised patient-level data, but does require a suitable framework for data access for the purpose of academic, non-commercial research. While the release of summarized data does not fully comply with the journal requirements, we welcome that the authors are now willing to share summarized data, and we will provide this once it is made available to us.
In spite of requests to the authors and Queen Mary University of London, we have not yet received confirmation that an institutional process compatible with the existing PLOS data policy at the time has been developed or implemented for the independent evaluation of requests for data from this study. We conclude that the lack of resolution towards release of the dataset is not in line with the journal’s editorial policy and we are thus issuing this Expression of Concern to alert readers about the concerns raised about this article.
Alongside the journal’s EOC is a strongly worded critique from the authors of the study, including last author Peter White at Queen Mary University of London, UK:
We disagree with the Expression of Concern about our health economic paper that PLOS ONE has issued and do not accept that it is justified. We believe that data should be made available and have shared data from the PACE trial with other researchers previously, in line with our data sharing policy. This is consistent with the data sharing policies of Queen Mary University of London, and the Medical Research Council, which funded the trial. The policy allows for the sharing of data with other researchers, so long as safeguards are agreed regarding confidentiality of the data and consent as specified by the Research Ethics Committee (REC). We have also pointed out to PLOS ONE that our policy includes an independent appeal process, if a request is declined, so this policy is consistent with the journal’s policy when the paper was published.
During negotiations with the journal over these matters, we have sought further guidance from the PACE trial REC. They have advised that public release, even of anonymised data, is not appropriate. As a consequence, we are unable to publish the individual patient data requested by the journal. However, we have offered to provide key summarised data, sufficient to provide an independent re-analysis of our main findings, so long as it is consistent with the REC decision, on the PLOS ONE website. As such we are surprised by and question the decision by the journal to issue this Expression of Concern.
A spokesperson for PLOS told us this is the first time the journal has included a statement from the authors in an EOC:
This has been a complex case involving many stakeholders and we wanted to document the different aspects of the case in a fair manner.
He pointed to this post on the PLOS site today, by the editor and managing editor of PLOS ONE, which explains further about the journal’s decision to issue the EOC, and the importance of data sharing.
We asked if the journal plans to retract the paper if the authors fail to provide what it’s asked for; the spokesperson explained:
At this time, PLOS stands by its Expression of Concern. For now, we have exhausted the options to make the data available in accordance with our policy at the time, but PLOS still seeks a positive outcome to this case for all parties. It is our intention to update this notice when a mechanism is established that allows concerns about the article’s analyses to be addressed while protecting patient privacy. PLOS has not given the authors a deadline.
One of the readers who has requested the data is James Coyne, a psychologist at the University Medical Center, Groningen, who submitted his request 18 months ago (and wrote about it on the PLOS blog site). Although some of the data have been released (to one person under the Freedom of Information Act), it’s not nearly enough to conduct an analysis, Coyne told us:
This small data set does not allow recalculation of original primary outcomes but did allow recalculation of recovery data. Release of the PLOS data is crucial for a better understanding of what went on in that trial. That’s why the investigators are fighting so hard.
Eventually, Coyne began suggesting to PLOS that he would organize public protests and scientific meetings attended by journal representatives.
I think it is the most significant issue in psychotherapy today, in terms of data sharing. It’s a flagrant violation of international standards.
That sentiment has been echoed in a 2015 STAT column by our co-founders Ivan Oransky and Adam Marcus, who noted:
If the information Coyne is seeking is harmful and distressing to the staff of the university — and that’s the university’s claim, not ours — that’s only because the information is in fact harmful and distressing. In other words, revealing that you have nothing to hide is much less embarrassing than revealing that you’re hiding something.
“Adaptive Pacing, Cognitive Behaviour Therapy, Graded Exercise, and Specialist Medical Care for Chronic Fatigue Syndrome: A Cost-Effectiveness Analysis” has been cited 19 times since it was published in 2012, according to Clarivate Analytics’ Web of Science, formerly part of Thomson Reuters.
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
Thanks for covering this. David Tuller has just had a piece appear in the Journal of Health Psychology about the problematic ways in which the PACE team respond to concerns about their work, and it seems relevent: http://journals.sagepub.com/doi/full/10.1177/1359105317703788#.WQHuZsKSA7Y.twitter
On this PLoS issue, they again appear to be avoiding presenting any sort of detailed argument as to why the specific, anonymised, requested data could not be shared in a way that accords with PLoS’s policies without breaching the trial’s consent forms. When they lost their case at an Information Tribunal, they had tried to claim that the trial’s consent forms somehow only allowed data sharing with those researchers they were collaborating with, and their arguments were rejected, despite the £250k (inc VAT) spent on lawyers. http://www.centreforwelfarereform.org/news/major-breaktn-pace-trial/00296.html
The limited data that they were forced to release has already been used to illustrate how the PACE team’s claims about recovery were misleading: http://www.tandfonline.com/doi/abs/10.1080/21641846.2017.1259724?journalCode=rftg20
It is clear that the £5 million+ PACE trial has considerable institutional support within the UK, where patient critics of the trial faced years of smears and stigma that allowed their concerns to be dismissed out-of-hand. I think that it is important that, as the problems with this research come to be more widely acknowledged, there is recognition of the fact that they reflect not just the failings of a few researchers but also of the systems of oversight and scrutiny that are supposed to be in place within UK science.
It also speaks to a wider problem in science in that a null result is seen as a failure, particularly where a lot of money has been invested. The result of the PACE trial would have been of no less interest had been been presented thus: ‘on any reasonable definition of recovery, talking and exercise therapies have been shown to be ineffective and thus further investigation is needed to find other strategies for dealing with this condition.’ Read that way, it would have been a valuable contribution to the science and medical treatment of ME/CFS.
However, headlines are not achieved and careers are not made on null results – and that is a more fundamental issue for science as a whole.
The wider push to get exciting results within science, and the way this can lower standards, is important, but in this case I think that the history of the PACE trial’s researchers in promoting their treatments, along with the way that they had already tried to dismiss patient concerns about their earlier work, meant that they had unusually strong personal incentives to hype results anyway.
It was a mistake to give funding for a supposedly “definitive” trial to a group of researchers who were personally invested in finding a particular answer. When he was promoting PACE, and dismissing the trial’s critics, Lancet editor Richard Horton presented things like this:
“I mean adaptive pacing therapy essentially believes that chronic fatigue is an organic disease which is not reversible by changes in behaviour. Whereas cognitive behaviour therapy obviously believes that chronic fatigue is entirely reversible. And these two philosophies are kind of facing off against one another in the patient community and what these scientists were trying to do is to say, ‘Well, let’s see. Which one is right?’”
http://www.abc.net.au/rn/healthreport/stories/2011/3192571.htm
In the commentary that accompanied PACE in the Lancet announced who had won the simplistic ideological battle Horton imagined was at the heart of things:
“PACE used a strict criterion for recovery:
a score on both fatigue and physical function within
the range of the mean plus (or minus) one standard
deviation of a healthy person’s score. In accordance with
this criterion, the recovery rate of cognitive behavioural
and graded exercise therapy was about 30%”
http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(11)60172-4.pdf
Unfortunately, what was put forth as a “strict criterion for recovery” was in fact so loose that patients could decline from baseline one every one of the trial’s outcome measures and yet still be classed as ‘recovered’. It is also not true that they had used the mean +/- 1sd of a health person’s score. In trying to justify not correcting this piece The Lancet itself has put forth a number of misleading arguments.
This comment is getting very long. Had the results from the PACE trial been presented fairly and accurately at first then that would have been better for everyone, but even then, the problems with the design of the trial and the history surrounding the PACE trial’s researchers would have meant that they, and those who had decided to fund the trial, would have received considerable criticism. I suspect that this helps explain some of what has happened since then.
Thank you for covering this important topic. But as an ME patient, I must point out that our objection is not primarily to the potential harms caused by exercise therapies in a disease defined by exercise intolerance, or a form of CBT which advocates we ignore our symptoms to hopefully make them go away.
Rather our objection is to the poor methodology used in the trial: there were no primary objective outcomes, the recruitment criteria are internationally discredited, and recovery criteria were weakened so much that patients could have deteriorated and still been counted as recovered.
Now that some recovery data has been released by a FOIA tribunal (which coincidentally rejected the authors’ recently rehashed claims of the data being too personal to release), we have been able to see that the authors’ changes had inflated recovery rates by a factor of 4. And even the minor gains which remained are not statistically significant.
This brings up another very important question, which Coyne and a great many patients, charities, and researchers would like to see answered: Is the PACE authors’ conclusion in the cost-effectiveness paper warranted? And certainly NICE, the NHS, and those responsible for them should be interested in knowing if these therapies are cost-effective in ME/CFS. Since the PACE authors clearly will not analyze their data based on their own protocol, this publicly-funded data needs to be released so that other scientists and academics can do so.
It is profoundly amusing that the detailed results of activities spending millions of taxpayers’ money were deemed “inappropriate” to be revealed, so that none of the outsiders could possibly know whether the money is well spent, if spent at all. As for this PLoS article, we should have absolutely no confidence in it as long as the full data could not be accessed openly. Actually, some people might even enjoy such articles on tabloids.
As the Director of the National Archive on Computerized Data on Aging at the University of Michigan, we have 35 years of experience that has shown that all data can be shared safely if appropriate protocols are used. The concern as stated by the PACE representatives are problematic at best, and distressing under and circumstances.
I applaud PLOS for issuing the EoC. This is fully in line with their policies on open data access, which a retroactive (apply to already published papers), progressive and much needed. Anonymize data clinical should be made available to other researchers. Anonymization is a trivial matter and the PACE researchers must make this anonymize data available in a timely manner or PLOS must retract.
PLOS must also apply their policies uniformly across their journals including PLOS Pathogens.
What is actually complex about this case? A reanalysis of the data carried out in accordance with the authors original trial protocol demonstrated just how misleading the authors claims are. How can the cost effectiveness of a ‘treatment” regime be determined when people could enter the trial get worse on 2 of the outcome criteria and be counted as recovered?? Objective data was collected…where are the results? 6 minute walk test, 2 minute walk test, heart rate, VO2 max, up and go test…..how did the outcomes of the patients with mental health disorders compare to those without? The EoC is good news. The right of reply is unusual and doesn’t shed any light on the issues at stake. We know that PACE were happy to share data with authors on the same panel as themselves such as the Cochrane review panel…and guess what when the PACE authors “marked” the PACE trial they marked it as a PASS. The detailed and severe flaws and faults were not addressed. Thank your Plos One for the EoC – I’m looking forward to the retraction unless QMUL release the full data set. 5 million pounds of tax payers money into the study and 250 000 pounds spent by QMUL attempting to NOT release the data requested by Alem Mathees FOI.
Thanks, PLOS One.