The Cochrane Library has withdrawn a criticized 2014 meta-analysis about a technique to help young people avoid alcohol abuse, because of “major errors.”
The review found that motivational interviewing, a form of counseling to help people change behaviors, showed some effects but had “no substantive, meaningful benefits” in preventing alcohol abuse among people 25 and younger. However, other researchers in the field, including some whose studies were included in the analysis, soon raised concerns about the review’s methods and data calculation, and the authors withdrew it.
Here’s the brief notice for “Motivational interviewing for alcohol misuse in young adults:”
Major errors in the Review have been identified through the feedback mechanism. The Review will be withdrawn until it can be replaced by a new version.
David Foxcroft, a professor at Oxford Brookes University and the paper’s corresponding author, told us he believes the issues stem from mistakes that are being corrected:
We made some data entry mistakes and we were alerted to this by other researchers. We therefore withdrew the review straight away and are fixing the data entry errors. The revised review will be re-published in the near future.
In August, three researchers in the field — Eun-Young Mun, David Atkins and Scott Walters — published a strongly worded critique of the Cochrane paper in the journal . The article, “Is Motivational Interviewing Effective at Reducing Alcohol Misuse in Young Adults? A Critical Review of Foxcroft et al. (2014),” seemed to flag more than data entry mistakes:
The present article critically reviews the Foxcroft et al. meta-analysis, highlighting weaknesses such as problems with search strategies, flawed screening and reviews of full-text articles, incorrect data abstraction and coding, and, accordingly, improper effect size estimation. In addition, between-study heterogeneity and complex data structures were not thoughtfully considered or handled using best practices for meta-analysis. These limitations undermine the reported estimates and broad conclusion made by Foxcroft et al. about the lack of MI effectiveness for youth. We call for new evidence on this question from better-executed studies by independent researchers.
Research conducted by Walters and Mun was included in the review; during a phone interview, both told us this fact helped them quickly recognize the alleged problems with the paper.
Mun, Atkins and Walters also emailed us a detailed statement explaining their side of the story, which you can read in full here. In it, they say the review made a “big splash” when it appeared in 2014:
Immediately, we felt its impact. We could also foresee major implications for young adults seeking treatment, MI trainers, clinicians, college administrators, intervention trial researchers, journal editors, funding agencies, and other stakeholders because it was incredibly dismissive of the beneficial effects of MI for alcohol misuse among young adults, and seemed to contradict the bulk of research to date. So, we decided to look at their data and conclusion carefully.
The full report included all the raw aggregated data from individual trials, so the authors performed their own analysis:
We soon discovered a number of errors in data abstraction and calculation. At first, we discussed whether to ask for corrections from some of the original researchers. However, because the errors seemed to be extensive and also because the review was flawed from early stages, it dawned on us that to simply alert errors may not sufficiently address our concerns.
They then sent their concerns to Cochrane’s editor:
When our critical review paper was accepted for publication, we contacted David Tovey, Editor in Chief of the Cochrane Library in June 2015. He recommended that we use their online feedback system to communicate our concerns with the Cochrane Drugs and Alcohol Review Group (Editors: Laura Amato and Marina Davoli). As soon as the critical review paper became available online on August 5, 2015, we submitted a short commentary, as well as the PDF review paper, via e-mail. The editors of the Cochrane Drugs and Alcohol Review Group contacted David Foxcroft for his response. He agreed on September 1, 2015 that it would be best to withdraw the review until he could address the concerns raised by us and others (see for instance, Grant et al., 2015).
Mun, Atkins and Walters noted they don’t suspect anything “intentional” on the part of the researchers:
We would like to note that although there were a series of questionable decisions and errors, as well as generally careless execution and presentation, we do not think that the review by Foxcroft and colleagues “intentionally” misled readers. From our standpoint, we also feel that Cochrane editors were responsive when we approached them with our concerns.
However, they noted that “damage may have already been done,” as the review prompted news stories such as from Reuters reporting that the counseling approach had few benefits.
In the future, it would be desirable to seek input from various stakeholders before new Cochrane review protocols are considered and before Cochrane reviews are published. Engaging and seeking input from stakeholders, especially those in the field developing and refining interventions when evaluating interventions, may promote better clinical practices and strategies. We need more concerted efforts across the data life cycle (from trial to input data for synthesis) to better understand the effectiveness of MI interventions and their mechanisms for heterogeneous young adult populations with different needs.
We sent the specific criticisms of Mun, Atkins and Walters to Foxcroft for his response. He told us he’s planning to include the three authors in the revisions:
…we have been updating the review to include some new studies (we had extended the search to bring it up-to-date to July 2015) and have corrected a few coding errors pointed out to us by other researchers (this was the reason we withdrew the review – to correct the mistakes). The revised and corrected review will be finished in 2-3 weeks and will be submitted via the Cochrane quality assurance process. I’ll also circulate to the authors of the critique at this time, as a confidential draft, for their comments and feedback. At the same time I have been working with Cochrane editors and statisticians to prepare a response to the other criticisms of the review.
We’ve asked Cochrane for a statement and we’ll update if they respond.
Update 4/5/16 2:40 p.m. eastern: Foxcroft has posted a response to the criticisms of the review, “We cannot ignore bias, especially if effects are small, but we need better methods for evaluating prevention systems,” in the journal Addiction:
We made a few data extraction and coding mistakes in our original review, so we asked the Cochrane Group to withdraw the review while we corrected these mistakes and updated the search. The updated review will be available in the first part of 2016, and it is unlikely that our original findings and conclusions will change.
Foxcroft addresses the individual criticisms from Grant et al — the last one touching on a larger issue in research:
The fourth point should be discussed more widely among researchers and policymakers: what sort of effect size is good enough? Our interpretation of the effect sizes we were finding across a series of meta-analyses was that they were very small, and unlikely to be of any meaningful benefit on their own, regardless of any possible but unknown reductions in effect sizes due to bias. For example, in the original review, we estimated that the obtained effect sizes would mean (approximately, on average): a decrease in the number of days per week that alcohol was consumed from 2.7 days to 2.5 days; a decrease in the number of drinks consumed each week from 13 to 11 drinks per week; and a decrease in the 69-point Rutgers Alcohol Problems Index (RAPI) from a score of 8.9 to 8.7. We suggest that these achieved effect sizes may fall short of a minimally important clinical difference (MCID). Grant et al. disagree with our interpretation, and that is alright: users of research evidence should make up their own minds on the right interpretation of the evidence.
Update 7/18/16 10:51 a.m. eastern: We’ve received an update from Foxcroft:
We have revised the review and it has been through an extensive quality assurance process. The original conclusions are unchanged and in fact are strengthened in the revised version. The new version has been accepted for publication in the Cochrane database and is available at http://onlinelibrary.wiley.
com/doi/10.1002/14651858. CD007025.pub4/full.
Hat tip: Daniel Mullins
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post. Click here to review our Comments Policy.