Note: This story has been updated to include the journal’s response. See below.
Yesterday, John Bohannon described in i09.com how he successfully”created” health news — he conducted a flawed trial of the health benefits of chocolate, gamed the data to produce statistically significant results, and published the findings in the International Archives of Medicine:
It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.
Given that the author himself says the study is meaningless, clearly, the journal will retract it, yes?
Not necessarily, given what we’ve seen in the past. Yes, Bohannon provided a false first name (“Johannes”) and affiliation, but to many journals these would be simple corrections; we’ve seen far worse “errors” that don’t rise to the level of retraction. No, he and his colleagues never received approval from an Institutional Review Board to conduct their research on human subjects, but Bohannon told Retraction Watch the journal never asked him to do that. While failing to document IRB approval that you say you’ve gotten, or lying about it, is certainly a reason for retraction, Bohannon isn’t guilty of either of those things. (We’ve contacted the journal’s editor in chief for comment, and will update with anything we learn.) Update 5/28/15 2:46 p.m. eastern: The journal appears to have pulled the paper from its website. Update 5/28/15 3:37 p.m. eastern: The journal’s Facebook page includes a statement from publisher Carlos Vazquez claiming the paper was published by mistake and was only live for a few hours:
Disclaimer: Weeks ago a manuscript that was being reviewed in the journal “Chocolate with High Cocoa Content as a Weight-Loss Accelerator” appeared as published by mistake. Indeed that manuscript was finally rejected, although it went online for some hours.
We are sorry for the inconvenience. We are taking measures to avoid this kind of mistakes happens again.
Unfortunately, here at Retraction Watch we see many flawed studies, whether intentionally so or not; that’s not going to change anytime soon. So how should we deal with such problematic research? In Bohannon’s case, the media ran with his findings without conducting any of the simple due diligence that would have uncovered the major issues. And that, to us, is where the meat — or chocolate — of this story is. Here’s Gary Schwitzer, publisher of HealthNewsReview.org, on why that really matters:
John Bohannon claims he “fooled millions into thinking chocolate helps weight loss.”
But he may have directly fooled only a few – not millions. And those few – whom I will politely call “journalists” – did the rest of the fooling for him.
And they do it all the time, gobbling up crumbs from a steady diet of weak, hyped studies.
When you critique (and try to improve) health news every day, as we have on HealthNewsReview.org for more than 9 years, you find broad, deep problems with the way science is communicated to the public.
And it’s not just on “terrible science” that is “fairly typical for the field of diet research,” as Bohannon frames the problem. It happens in stories about cancer, heart disease, diabetes – you name it. And then we really are impacting millions.
Here are just a few of the notes I took from Bohannon’s article on io9.com:
…for reporters who don’t have science chops, as soon as they tapped outside sources for their stories—really anyone with a science degree, let alone an actual nutrition scientist—they would discover that the study was laughably flimsy.
Well, he found out how likely outside sources were to be tapped.
Not a single reporter seems to have contacted an outside researcher. None are quoted.
This isn’t a surprise to me. Only about half of the 2,000 news stories we’ve reviewed received satisfactory grades for using independent sources and for identifying conflicts of interests in sources.
You can’t absolve journals of some of the responsibility – and not just the “fake journals” Bohannon has targeted with his stings. He writes:
Almost no (journal) takes studies with fewer than 30 subjects seriously anymore.
I beg to differ. Here’s a chocolate-related nutrition journal article on a study of just 24 women. Topic: a particular kind of antioxidant-rich cocoa protects the skin from sun damage caused by UV exposure. Really? The Minneapolis Star Tribune, one of the biggest newspapers in the country, posted a fluff piece on it on their website.
We’ve found that it’s not unusual for reputable journals to publish papers on studies with fewer than 30 subjects. And that’s certainly not the only — or the biggest flaw — in what journals publish.
Talk about cancer, and some journalists will hint at “cure” even when reporting on a study of 22 people, half of whom died. And they don’t need a journal article or a news release prompting them.
You don’t have to create a spoof to lure journalists. Chocolate claims of any sort seem to work every time.
We also reviewed a Mars , Inc. (the chocolate company) news release that claimed “mounting evidence demonstrates improved cognitive function from cocoa flavanol consumption.” Our reviewers said that the release did “not include a single number related to the benefits, and notes merely that there was ‘significant’ improvement with some dosages of flavanols.” Devoid of data. Even so, some news stories bit the bait. This is similar to the Bohannon spoof, except it wasn’t a spoof and it came from a chocolate company. An unholy wedding of marketing, public relations and journalism.
We don’t even have to leave home to do any reporting. We just dip our cups into the daily stream of scientific press releases flowing through our inboxes.
Even esteemed medical journals’ news releases are complicit in the chocolate hype.
We criticized a BMJ news release for sensationalizing an observational study with the headline, “It’s official: chocolate linked to heart health.” Many news organizations followed like sheep. NBC News was laughable with lines such as “Bona fide science. A real study of a lot of people.”
It was a university news release that led influential news organizations to report that “a new study indicates” that “drinking iced tea raises kidney stone risk.” Only problem is, there was no study.
With that as a backdrop, should we really be surprised to see journalists fooled by tricks related to statistical significance, as Bohannon writes? P-values? P-hacking?
When we have journalists reporting on studies that never existed?
When we have news releases from industry, academic medical centers and medical journals misleading journalists who then mislead the public?
I don’t know if Bohannon’s latest stunt will have any positive impact. Journalists and bloggers have already moved on to the next, much-quicker-than-24-hour, news cycle. Here at HealthNewsReview.org, we are doing our best to help readers see through the hype of health news. By publishing reviews continuously, we hope to help more people understand, do a better job communicating, and make better decisions.
The health and science news food chain that feeds the consuming public is contaminated by different players at different stages along the way. The problem is much worse than a publishing prank that fools millions. It’s a credibility crisis for science and for journalism. And consumers at the end of the food chain will be – indeed, already are – poisoned.
Gary Schwitzer is the publisher of HealthNewsReview.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.