Following revelations in io9.com this week from John Bohannon about how he successfully “created” health news by conducting a flawed trial of the health benefits of chocolate and gaming the data to produce statistically significant results, the journal that ultimately published the findings is now claiming the paper wasn’t accepted.
Trouble is, we’ve got correspondence from Bohannon showing that’s false. Here’s a quote from an email from publisher Carlos Vazquez to which Bohannon responded on March 2:
I’m contacting to let you know your manuscript “Chocolate with High Cocoa Content as a Weight-Loss Accelerator” has been pointed by our editors as an outstanding manuscript and could be accepted directly in our premier journal *International Archives of Medicine.*
The Facebook page for the International Archives of Medicine includes a statement from Vazquez claiming the paper was published by mistake and was only live for hours:
Disclaimer: Weeks ago a manuscript that was being reviewed in the journal “Chocolate with High Cocoa Content as a Weight-Loss Accelerator” appeared as published by mistake. Indeed that manuscript was finally rejected, although it went online for some hours.
We are sorry for the inconvenience. We are taking measures to avoid this kind of mistakes happens again.
Although the journal appears to have pulled the paper from its website, it was there as of this morning (we saw it). Given that the paper was published in April, that’s significantly more time than “some hours.”
Bohannon tells Retraction Watch:
They took our money, they published it online, and it was published for weeks, so enough said.
Asked if they would be requesting a refund, he said:
If they do make the paper disappear, we will ask for the money back, hell yeah.
This isn’t Bohannon’s first sting operation — in 2013, he found that fewer than half of publishers of open access journals rejected deeply “flawed” papers which he assembled.
Like Retraction Watch? Consider supporting our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.
“We are taking measures to avoid this kind of mistakes happens again.”
The International Archives of Medicine, where nothing can possibly go worng.
The editor serves as our Grauniad against errors.
Yahoo’s cached copy of the web page about the paper:
http://www.burtonsys.com/Chocolate_with_high_Cocoa_content_as_a_weight-loss_accelerator.html
Google’s cache of the paper (pdf converted to html):
http://www.burtonsys.com/Chocolate_with_high_Cocoa_content_as_a_weight-loss_accelerator2.html
A couple of web articles, by authors who took the bait:
http://www.infinitelabs.com/benefits-of-dark-chocolate/
http://suppversity.blogspot.com/2015/03/dark-chocolate-diet-booster-for-low.html
And Google finds many more:
https://www.google.com/search?q=%22Chocolate+with+high+Cocoa+content+as+a+weight-loss+accelerator%22
You have to hand it to Bohannon. Regardless that his methods are somewhat ethically questionable, he’s doing a pretty good job (way better than any of us) right now, of handing science publishing and journalism their collective asses on a plate. I’d like to see more of this type of thing… Anything that gets the public and scientists to realize what a sham the whole science publishing game is, is a good thing.
The article was definitely still available yesterday afternoon (AEST) as I sent out the link via Twitter. This morning the PDF has disappeared.
However, apart from the publisher’s behaviour there are also some serious questions about Bohannon’s ethics: In his post on io9 he claims that this “operation” was performed in the spirit of “Gonzo” journalism. I am not sure if Hunter S Thompson would have called misleading study participants “Gonzo” journalism, as the investigator hardly put himself in the line of fire. Also, the reason I went to check on the actual paper was the question if it was ethical to publish this in the first place. One glaring omission in the performed work is the apparent lack of consent from the study participants, and/or IRB approval of the study. The questions that have to be asked are:
“Is it ethical to perform research involving human participants without consent”? (No, it is not. It is actuall illegal in most countries).
“Is it ethical to involve volunteers in a study which was performed with the sole purpose of producing fake data?” (No, it is not. Ask yourself: would you volunteer for a study which express purpose was to fake data?)
This is not to say, that Bohannon did not have a valid reason for shining the light on the way media handles announcements of health research, in particular when it comes to dieting. However, he might have just as well made all of the data up, invented his co-authors, including the MD who did the actual surveys and probably would have still been able to find a pay-to-publish OA outlet to get a “peer-reviewed” paper out of it. Hell, I get dozens of invitations a week to publish in obscure journals which promise 48 hour turnaround on peer-review and rapid publication…
I wish RW would consider these points as well, instead of just aiming fire at the evil, lying publisher.
Have to make a correction to the last sentence:
RW did consider the questions of unethical behaviour by the authors of the study. Unfortunately this was lost on me in the flurry of updates/quotes/notes in the first paragraphs of the previous post:
http://retractionwatch.com/2015/05/28/should-the-chocolate-diet-sting-study-be-retracted-and-why-the-coverage-doesnt-surprise-a-news-watchdog/
The data wasn’t “fake”, Bohannon actually did recruit patients, they did weigh themselves every day, the experiment really was performed, and the results were reported honestly. The study was designed to create a statistical fluke, which is normal for poor-quality studies. One does not need to explicitly fake data to get the desired result, that’s what this exercise was designed to show.
“Is it ethical to involve volunteers in a study which was performed with the sole purpose of producing fake data?” (No, it is not. Ask yourself: would you volunteer for a study which express purpose was to fake data?)
The participants knew that the express purpose of the exercise was not to collect meaningful results, but to film a TV program. Yet they volunteered.
Both the Declaration of Helsinki and the Nuremburg Code state that the fundamental principle of ethical (medical) research is around respecting the rights of individuals to make voluntary, informed decisions about their participation in research. Research involving deception usually attracts a higher level of scrutiny by medical ethics committees since it impacts on informed consent, and could increase the risk of mental harm to participants.
Bohannon post – http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 – says:
“They [the documentary makers] used Facebook to recruit subjects around Frankfurt, offering 150 Euros to anyone willing to go on a diet for 3 weeks. They made it clear that this was part of a documentary film about dieting, but they didn’t give more detail.”
This already would seem to involve deception, since the true nature of the study (exposing p-hacking and inadequate peer-review) was apparently not disclosed.
Potential participants were then screened by the General Practitioner, Gunther Frank, using questionnaires and blood tests. Again, there’s nothing in the post to suggest participants were informed of the true reason for the “clinical trial”.
I’m wondering if participants signed any sort of consent or waiver as part of the documentary? And what protections are in place for their clinical / medical / health data which Frank, the GP, collected?
Dieting ………. the whole thing isn’t worth discussing, is it?
To spend time doing this kind of paper is absurd, unless the person has an interest, either promoting himself or ‘down’ promoting others (publisher). Anybody with the right skill would know whether a ‘published’ paper is fake/dubious/etc.
Such people are weeded out before they become journalists.
Abject apologies to any *real* journalists who not only do their jobs, but also set up websites to scrutinise the failures of science journalism.
I’m not saying the journal and it’s going ons aren’t questionable, but when I receive a letter from an editor saying my paper “could” be published I don’t interpret that as “is”. Not yet.
I wonder what would happen, apart from most likely an immediate rejection, if Bohannon would submit a similar “sting” to an Elsevier, Springer, Wiley or Taylor and Francis journal?
Any study involving human participants (particularly involving invasive procedures such as blood tests, and deceptive techniques such as concealing the purpose of the study) involves weighing up the risks of the study to the participants, against the wider benefit to society of doing the study. Whether the study is ethical depends on the balance of these risks and benefits. Who should get to decide on this balance, and whether the study should be allowed to happen? It’s not you, me or the person running the study – we have review boards specifically to do this job. In my view bypassing those review boards is unethical.
If it so happens that we as a society disagree with the balance that the review boards decide, then we should lobby to change the criteria that the boards use, not bypass them.
I can’t see any excuse for not getting ethical approval for this study.
Tim, could you suggest exactly from which review board ethical approval should have been obtained, i.e., of which author? I am not sure of the affiliation that Bohannon used, so could someone indicate this please. Also, what are the affiliations of the co-authors, and what are their academic qualifications? Even though Bohannon has a PhD, and even though he has experience with “stings”, does this qualify him to be the lead author of a “medical” “study”?
In my post I think I assumed that this study was affiliated to a university, in which case I would imagine the university would have guidelines. Outside of a university I don’t know what the rules would be.
As I understand it, all author names and the affiliation are fake. The lead author on a publication does not have to be the same as the chief or principal investigator for the study, so in that sense it’s fine for Bohannon to be first.
The issue of ethics is interesting. According to the io9 story, Gunther Frank is the General Practitioner who “ran the clinical trial”. It isn’t clear if that’s a real or fake name. The (Model) Professional Code for physicians in Germany has a section on research (see pp 13-14), English version here:
http://www.bundesaerztekammer.de/fileadmin/user_upload/downloads/MBOen2012.pdf
This suggests ethical review should’ve happened to cover Frank’s involvement. I’d suggest the specific board would be one associated with Frank’s employment.
Paul Brookes: “I’d like to see more of this type of thing…” Among all of the smart things you have ever proclaimed, this was certainly not one of them. It is unacceptable, in any manner or form, to encourage any dishonest behaviour in science publishing. Your encouragement should be strongly rebuked, because in essence what you are saying is increase the number of stings (which include fake or false information, potential breaches of ethics, and possible deceipt) to achieve a point. You don’t have to cheat journals, and waste precious human time and resources to make a point: you can just start your own blog, like Beall did. I am of the opinion that Bohannon reverts to incorrect, and possibly unethical, means, to achieve his objectives. No matter what the learning curve turns out to be, the methodology to achieve it, as I see it, must never be praised, approved, or encouraged. If a person wants to show their anger, rage or disappointment, go ahead, but please don’t cheat the system, which is already under sufficient strain.
Here’s a study goes to 11!
Similar to Johannes et al, physiologist Brent Ruby, University of Montana, Department of Health and Human Performance was dubious of the rigor of the science behind the claimed benefits of sports supplements. In a report from High Country News, Ruby said the makers of sports supplements tend to exaggerate their effectiveness, touting their clear “natural” superiority to something as obviously odious as a double cheeseburger with an extra serving of fries. reported that “Ruby devised an experiment involving 11 active men who cycled for 90 minutes and were tested before and after consuming either the sports supplement or fast food from McDonald’s, which precisely measures its servings and was conveniently located just across the street. The result: Muscle recovery and exercise performance “were not different when comparing products created specifically for sport recovery and traditional fast food.” http://www.hcn.org/issues/47.9/a-prom-request-painted-on-blm-land-a-nevada-brothel-and-more
So n=11 is as good for supporting McDonald’s for athletes as n=16 for supporting chocolate for weight loss!
In contrast to Ruby’s puckish study, Johannes was ethically dubious at best to tell people they were part of a TV program about weight loss, take intrusive blood draws, when the real intent was to poke the junk-science diet industry. I cheered Bohannon for his sting of the sketchy vanity/predatory open access journals, despite its whopping flaw (no control submissions to “traditional” journals). To me, in the the present exposé, it’s questionable whether the ends justified the means.
By the way, did anyone snag the actual Bohannon PDF before it disappeared? The Google archive converted to HTML seemed incomplete. For instance, the iO9 website said n=16, but I didn’t see that in the article. Maybe that was the point, not mention the small size and see if the “reviewers” even bother to ask.
A perfect critique by Hilda Bastian here:
http://blogs.plos.org/absolutely-maybe/tricked-the-ethical-slipperiness-of-hoaxes/#comment-12966
Uploaded here: https://commons.wikimedia.org/wiki/File:Chocolate_with_high_Cocoa_content_as_a_weight-loss_accelerator.pdf
I totally love it how open-access-journals (especially those with a creative commons license) really CANNOT retract any paper. Everybody can re-publish the papers, and point to the original journal with a certain amount of glee and schadenfreude.
But, of course, that’s not an argument against open access.
Turtle, not sure who uploaded that file onto Wikimedia, but it’s a work of art, for posterity to judge. Excellent job, whoever did it.