The week at Retraction Watch featured controversy over an economics paper, and a report of a researcher who faked more than 70 experiments. Here’s what was happening elsewhere:
- The NFL “improperly attempted to influence” where funding for concussion research went, Ike Swetlitz reports at STAT based on a Congressional finding. Here’s more from Roger Pielke, Jr.
- What’s the craziest item on the Russian black market? It’s not oil or guns, says Leon Neyfakh in Slate: It’s PhD dissertations.
- “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” Monya Baker writes about a new survey by Nature.
- Communications officers at journals and elsewhere should publicize retractions better, says Denise Graveline.
- The story of the controversial 2010 “arsenic life” paper in Science shows the effect blogs and Twitter can have in post-publication peer review, according to a new analysis. (Public Understanding of Science, sub req’d)
- 100% open access scientific literature by 2020? That’s what European leaders are calling for, reports Martin Enserink at Science.
- “Why do so many studies fail to replicate?” asks Jay Van Bavel. (New York Times)
- “Should researchers publish their findings before peer review?” ask our co-founders in STAT.
- When it comes to sports science, women are often underrepresented in studies, reports Bethany Brookshire. (Science News)
- “[J]ust because a journal states that it publishes replication studies, doesn’t mean that it does it very often,” says Bob Reed. (The Replication Network)
- Why don’t researchers share their data? A few answers from Nick Brown.
- According to Google Translate, this Swiss National Radio segment (in German) calls Retraction Watch the “Garbage Collectors of Science.” We’ll take it.
- Roberta Villa of Pagina99 takes a look at peer review fraud and other kinds of scientific misconduct. (in Italian)
- Law schools worry about rankings based on citations. Brian Galle has a way to make that system work better.
- A group of researchers is pressing PLOS ONE for action on a 2012 paper reporting findings from the controversial PACE trial of chronic fatigue syndrome. (Virology Blog)
- A NASA analysis of asteroids is deeply flawed, says Nathan Myhrvold in a new preprint. (Ken Chang, New York Times) NASA has responded in a statement that claims peer review ensures reproducibility.
- Fraud Rounds: Our Ivan Oransky gave Medicine Grand Rounds at Weill-Cornell on Wednesday. Here’s his presentation.
- “Don’t Let Cancer Immunotherapy Hope Give Way To Hype,” writes Alaric DeArment. (Forbes)
- Want to crowdfund your research? Here are tips from Jens Hegg. (PLOS Blogs)
- Does the U.S.have a different scholarly publishing culture? So says Brad Fenwick. (Elsevier Connect)
- “What if academic and scholarly publishers paid research authors?” asks Kent Anderson. (Scholarly Kitchen)
- Press embargoes are making some researchers concerned that after their papers’ metadata appear online, they’ll lose the chance to publish, says the University of Cambridge Office of Scholarly Communication.
- On Slate‘s The Gist podcast, our Ivan Oransky explains why weaknesses in peer review don’t mean the anti-vaccine movement is right — and discusses his dog.
- From NPR: “When great minds think unlike: Inside science’s ‘replication crisis.'”
- “Newspaper can’t wait another hour to reveal 5,000-year-old beer recipe, breaks embargo.” From our sister site, Embargo Watch.
- “Until we develop the right system, we don’t need or want money back guarantees on research reproducibility,” says Eric Topol. (BMJ)
- Think your PhD years were tough? asks Connie Ruzich. This soldier finished his thesis in the trenches of World War I.
- “Salesmen say ‘Always be closing’, academics might say the same ‘Always be citable,'” says David Levinson.
- In a few top journals, “on average papers with shorter editorial delay do have larger probabilities of becoming highly cited papers,” according to a new study in Scientometrics. (sub req’d)
- How to get published: A guide from The BMJ‘s Trish Groves.
- In the future, Jeremy Garwood wonders if “the system of peer review that has risen over the last half-century will be remembered as a defunct model.” (Lab Times)
Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.
“Garbage Collectors of Science” (Müllsammler der Wissenschaft) is clickbaitish misleading. The report is more about elucidating background of retractions, and what can be learned from that (“Oransky hat es sich zur Aufgabe gemacht … die Hintergründe auszuleuchten”). In the report, “schnüffeln im Abfallskübel” is more accurate, but more difficult to translate (“sleuthing in waste basket”?), and definitely not as clickbaitish as Garbage Collectors of Science.
A great badge to wear, and maybe someone should order some buttons. I’ll take one.
I appreciate the clarification. “Garbage Collectors of Science” just does not in any away capture what RW has been doing for science. Thank you!
Sifting through other people’s garbage can actually be quite a respectable activity in some parts of Switzerland. In the 1990, some cantons (Swiss federal “states”), including Zurich, introduced a new way to collect taxes on waste disposal. Instead of paying a fixed tax per household, you have to buy special expensive garbage bags. The idea is that in that way, the tax you pay is proportional to the amount of garbage you actually produce. See for instance here for a detailed explanation:
http://www.englishforum.ch/daily-life/197248-waste-disposal-recycling-zurich-2014-comprehensive-overview.html
Of course, a number of people tried to circumvent the system, either by using non-standard bags, or by dumping their garbage in neighboring cantons that did not use this tax system. Hence the need for a “garbage police”, sifting through the illegal bags to try finding clues that would allow to identify the perpetrator.
So “Dumpster Divers of Science”?
Interesting, “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.”
This means that the vast majority of published science represents proverbial GIGO (garbage in, garbage out). One would wonder, what is the $ cost of this? Given that most of the research is funded by various government agencies, there is a lingering question; who is accountable for such gross mismanagement of public funds?
No. Rather, it’s a reflection of the complexity of most research studies (particularly in the biomedical sciences) and the incredibly difficult task to control and to report all the experimental variables that go into designing, conducting, and analyzing experiments.
Michael, the problem is that many researchers do not understand limitations of their data and are stretching the results in an attempt to find things that in reality do not exist. Some “scientists” outright fabricate the data.
There are MANY reasons for an attempted reproduction to fail, and only some of them mean the original was garbage. Others include failure to follow the protocol, misinterpreted protocol and badly written protocol. Bad writing is a bad thing but it is often perfectly innocent – not much effort goes into teaching writing, and many authors are not native speakers of the language they’re writing in.
The large number given for people who’ve failed to reproduce their own results is easily taken out of context. The survey did NOT say failed to reproduce their own PUBLISHED results. Checking for reproducibility is why good scientists do their experiments more than once and think hard about the results. Really good scientists figure out why different experiments are giving different answers, and often it reveals something unexpected and interesting.
And bear in mind: if your own attempt at souffle turns out to be a flat mess of gunk, is all of fine cuisine really fraudulent?
Yes there is garbage out there that needs to be illuminated (which is why I read this blog), but it shouldn’t be used to trash the overall scientific endeavor.
I fully agree. The survey is also rather vague, as it is not clear what someone would consider “irreproducible”.
Note also that “bad writing” does not even need to be involved in problems replicating. My own experience during my Master’s provides an excellent example. I used a procedure from the literature to prepare a very specific oxidized tryptophan species. For some reason I did not just get the intended product, but also another one that was not described, and in pretty large amounts. There we go, I was unable to reproduce the literature!
It took me some time to come up with a possible explanation, which was almost immediately dismissed by my supervisor: I had closed the reaction vessel, and the reaction involved an equilibrium with a gaseous compound, so this may have mattered. I had good reason to close the vessel, because that gaseous compound was dimethyl sulfide, which smells really, really bad in the ‘large’ amounts formed during the reaction. Fortunately I was stubborn enough to test my hypothesis and try with the reaction vessel open. I was now able to reproduce the findings, gone was this unwanted side-product!
Was it bad writing of the original literature that they did not mention they left their reaction vessel open? If they had not tested with the reaction vessel closed (which is very likely), they would also not have known this mattered. It would be one of those “trivial” things that we don’t think about, but that actually do matter. Like male vs a female animal handler: http://www.nature.com/news/male-researchers-stress-out-rodents-1.15106
“Garbage Collectors of Science.” We’ll take it…. I suggest Public Health Inspectors..
The lack of reproducibility is often due to a lack of statistical power. If studies are underpowered then there are two possible problems. If we obtain a significant result then it is likely to be an error, because if there is a real effect then the chance of detecting it is low unless the effect is huge. Now if an effect is observed then we are unlikely to find it in a replication if the effect is real because the experiment is underpowered.
So the answer is not getting rid of p-values, although thinking about evidence rather than rejection is a good idea. What is important is to run properly powered experiments. That means being honest about meaningful differences and variances. Psychology is among the worst and why they probably have the most problems with replication. Neuroscience is just horrible, use 10 per group but run with 40 outcomes so something will look good, but will never be replicated.