PLoS ONE has retracted a meta-analysis on mindfulness after determining that the authors used dubious methodology and failed to adequately report their financial interest in the psychological treatment the article found effective.
The article, “Standardised mindfulness-based interventions in healthcare: An overview of systematic reviews and meta-analyses of RCTs,” appeared in April 2015 and has been cited 130 times, according to Clarivate Analytics’ Web of Science, earning it a “highly cited paper” designation.
The authors, from Erasmus Medical Center in the Netherlands and Harvard University, included Herbert Benson, of the Benson-Henry Institute for Mind Body Medicine at Massachusetts General Hospital. The institute (which has taken down its link to the paper) offers a raft of services for patients, including a Stress Management and Resiliency Program, a Mind Body Program for Health and Fertility, a Mind Body Program for Cancer, yoga, Tai Chi and initiatives to help foster “resilient youth.”
The decision comes after a long effort by James Coyne, an emeritus professor of psychology in psychiatry at the University of Pennsylvania, to expose the undisclosed conflicts and other serious problems in the work and other studies with ties to Benson-Henry.
Coyne first contacted PLoS ONE with concerns in October 2015 after the journal published a paper titled “Relaxation response and resiliency training and its effect on healthcare resource utilization,” by Benson and several other researchers. As Coyne noted, the group neglected to disclose any financial conflicts, despite the fact that Benson and six of his co-authors worked at Benson-Henry, which generates revenue by selling products rooted in its mindfulness approach.
The journal, to its credit, quickly followed up on Coyne’s tweet and launched an investigation. And in 2017 it issued corrections for the relaxation article and four other papers by Benson’s group for misleading disclosures, as Coyne wrote on his blog. The statements now read:
The Competing Interests statement is incorrect. The correct Competing Interests statement is: The following authors hold or have held positions at the Benson-Henry Institute for Mind Body Medicine at Massachusetts General Hospital, which is paid by patients and their insurers for running the SMART-3RP and related relaxation/mindfulness clinical programs, markets related products such as books, DVDs, CDs and the like, and holds a patent pending (PCT/US2012/049539 filed August 3, 2012) entitled “Quantitative Genomics of the Relaxation Response.”
An ‘experimercial?’
Meanwhile, Coyne had turned his attention to the meta-analysis, which the journal had not corrected even though it had a similarly uninformative disclosure. In March 2017 he wrote about what he considered its manifold flaws. Among these, he said, were another misleading disclosure statement, Benson-Henry’s use of the paper as an “experimercial” to promote the institute’s wares (more on that in a bit), and that it
sidesteps substantial confirmation bias and untrustworthiness in the mindfulness literature.
He also pointed out that the academic editor on the paper, Aristidis Veves, works at Harvard Medical School — yet another potential conflict of interest.
After two years, the journal decided to agree officially. According to the lengthy retraction notice:
In this article [1], the authors presented a systematic overview of 23 systematic reviews and pooled results from the 8 (of the 23) reviews that reported meta-analyzed standardised mean differences (SMD) of five outcomes. After the article was published, it came to light that the handling Academic Editor shared an affiliation with three of the authors. Due to this potential competing interest, the PLOS ONE Editors had the article reassessed by another member of the journal’s Editorial Board. During this reassessment concerns were raised about pooling of meta-analytic results in the meta-analysis, i.e. the authors considered each meta-analysis as an individual study in the analysis, rather than pooling results of individual RCTs as per community standards for this type of study. Since some randomized clinical trials (RCTs) were included in more than one meta-analysis, this pooling resulted in double counting, incorrect effect estimates in Fig 1A, 1B and 1E, and incorrect confidence intervals.
The authors noted that they had set a double counting maximum a priori in the meta-analyses, and excluded reviews that reported duplicate data until the predefined limit (10% per outcome) was reached, as explained in the Methods section (S3 Table identified the double counted RCTs per pooled outcome).
On re-evaluation of the data post-publication, the authors found two additional errors. In the supplementary table the two RCTs from the Veehof et al. 2011 meta-analysis were inadvertently not listed and in Fig 1 the authors had inadvertently included in pooled estimates of anxiety the meta-analyses of Cramer et al. 2012 and Galante et al. 2012, which led to double-counting beyond the predefined maximum.
The study’s pooled estimates and confidence intervals for the outcomes stress and quality of life did not include any double counted results. However, this issue impacted the other results reported in the article, and the consulted Academic Editor advised that a full re-analysis would be needed, in which individual RCTs rather than meta-analyses are pooled, in order to correctly address the study’s aims.
In addition, we note here that the following information was missing from the article’s Competing Interests statement: Herbert Benson and Gregory Fricchione hold or have held positions at the Benson-Henry Institute for Mind Body Medicine at Massachusetts General Hospital, which is paid by patients and their insurers for running the SMART-3RP and related relaxation/mindfulness clinical programs, markets related products such as books, DVDs, CDs and the like, and holds a patent pending (PCT/US2012/049539 filed August 3, 2012) entitled “Quantitative Genomics of the Relaxation Response.”
In light of the methodological issue and concerns about the validity of the study’s results, the PLOS ONE Editors retract this article. We regret that these issues were not fully addressed prior to the article’s publication.
The journal added that none of the authors agreed with the retraction. We emailed the corresponding author, Maria Hunink, of Erasmus and Harvard, to ask why but have not heard back.
Exploitation by commercial interests?
Coyne tells us he wasn’t looking for a retraction when he raised his issues about the meta-analysis with PLoS ONE. Rather, he says, his beef is grounded in his belief that unscrupulous researchers are using — or misusing — the journal for commercial purposes:
I am very concerned about the open access and loose editorial control of PLOS ONE as a megajournal being exploited by commercial interests seeking a peer-reviewed article to advertise their products.
Coyne, whose behavior in other cases has led to significant criticism, has been a reviewer for PLoS ONE for many years, although he has refused to look at manuscripts for some time because of a long-running dispute with the journal over its handling of a controversial UK study called PACE. The authors of that study have declined fully to share their data, prompting PLoS ONE to issue an expression of concern in 2017 but not retract the article.
David Knutson, the senior communications manager for the journal, says:
In terms of competing interests, PLOS ONE requires authors to declare potential competing interests so this information is available during the peer review process and upon publication of the article. For all studies, our publication criteria require that conclusions are well-supported by data, and submissions are subject to a rigorous peer review process. If concerns about potential competing interests or research reporting come to light post-publication, we address them per guidelines of the Committee on Publication Ethics (COPE).
Like Retraction Watch? You can make a tax-deductible contribution to support our growth, follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up for an email every time there’s a new post (look for the “follow” button at the lower right part of your screen), or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Kudos to Coyne for quickly identifying problems and bringing them to the journal’s attention. Kudos to PLOS One for (eventually) addressing them all. And thank you to Adam for the write-up.
Although I’m not in a research field, I wonder whether a journal could assist readers with an indication (or flag) that there is an identified problem (in this case, initially with undeclared COI) that the journal agrees with and is under investigation.
In this case, it appears from Sarah Bangs’ reply to Dr. Coyne that the journal acknowledged way back on 10/28/15 that there was a problem, but readers of this paper would not official learn of it until 2017, as I understand the timeline. I am sympathetic to how long a thorough investigation may take, (especially if the authors are not in agreement or if the scope of the investigation grows), but I think the underlying problem is how the research community feels about criticism of their papers, including “editor’s notes” and the like. Journals are therefore reluctant to “flag” a paper until a solution to an acknowledged problem is agreed upon.
It seems (to an outsider like me) that scientists should welcome robust scrutiny and criticism of their published work and respond vigorously with evidence and appreciation for publication ethics. If the authors can successfully defend their papers from critics, the published work appears more valid and becomes more impactful. The way some react suggests that their scientific reputation, their papers, and their findings are such fragile creatures that they cannot survive any challenge. To me, that sign of weakness is a fundamental, and suspicious, flaw.
If a journal flags a paper wrongly, they could get their pants sued off. Reputations are indeed fragile, and valuable (nay, essential) for researchers. Absent formal findings to rely on, few publishers would risk taking any public position on a problematic paper at least until there was a critical mass of reports and evidence already out in the public sphere. To me, it doesn’t seem to me like it’s about finding a “solution” before flagging a paper, but rather the journal ensuring it’s rear end is well-covered before taking any action whatsoever.