Weekend reads: Improper influence by NFL; dissertations for sale; how common is failure to reproduce?

booksThe week at Retraction Watch featured controversy over an economics paper, and a report of a researcher who faked more than 70 experiments. Here’s what was happening elsewhere:

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.

11 thoughts on “Weekend reads: Improper influence by NFL; dissertations for sale; how common is failure to reproduce?”

  1. “Garbage Collectors of Science” (Müllsammler der Wissenschaft) is clickbaitish misleading. The report is more about elucidating background of retractions, and what can be learned from that (“Oransky hat es sich zur Aufgabe gemacht … die Hintergründe auszuleuchten”). In the report, “schnüffeln im Abfallskübel” is more accurate, but more difficult to translate (“sleuthing in waste basket”?), and definitely not as clickbaitish as Garbage Collectors of Science.

    A great badge to wear, and maybe someone should order some buttons. I’ll take one.

    1. Sifting through other people’s garbage can actually be quite a respectable activity in some parts of Switzerland. In the 1990, some cantons (Swiss federal “states”), including Zurich, introduced a new way to collect taxes on waste disposal. Instead of paying a fixed tax per household, you have to buy special expensive garbage bags. The idea is that in that way, the tax you pay is proportional to the amount of garbage you actually produce. See for instance here for a detailed explanation:
      http://www.englishforum.ch/daily-life/197248-waste-disposal-recycling-zurich-2014-comprehensive-overview.html
      Of course, a number of people tried to circumvent the system, either by using non-standard bags, or by dumping their garbage in neighboring cantons that did not use this tax system. Hence the need for a “garbage police”, sifting through the illegal bags to try finding clues that would allow to identify the perpetrator.

  2. Interesting, “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.”

    This means that the vast majority of published science represents proverbial GIGO (garbage in, garbage out). One would wonder, what is the $ cost of this? Given that most of the research is funded by various government agencies, there is a lingering question; who is accountable for such gross mismanagement of public funds?

    1. No. Rather, it’s a reflection of the complexity of most research studies (particularly in the biomedical sciences) and the incredibly difficult task to control and to report all the experimental variables that go into designing, conducting, and analyzing experiments.

      1. Michael, the problem is that many researchers do not understand limitations of their data and are stretching the results in an attempt to find things that in reality do not exist. Some “scientists” outright fabricate the data.

    2. There are MANY reasons for an attempted reproduction to fail, and only some of them mean the original was garbage. Others include failure to follow the protocol, misinterpreted protocol and badly written protocol. Bad writing is a bad thing but it is often perfectly innocent – not much effort goes into teaching writing, and many authors are not native speakers of the language they’re writing in.

      The large number given for people who’ve failed to reproduce their own results is easily taken out of context. The survey did NOT say failed to reproduce their own PUBLISHED results. Checking for reproducibility is why good scientists do their experiments more than once and think hard about the results. Really good scientists figure out why different experiments are giving different answers, and often it reveals something unexpected and interesting.

      And bear in mind: if your own attempt at souffle turns out to be a flat mess of gunk, is all of fine cuisine really fraudulent?

      Yes there is garbage out there that needs to be illuminated (which is why I read this blog), but it shouldn’t be used to trash the overall scientific endeavor.

      1. I fully agree. The survey is also rather vague, as it is not clear what someone would consider “irreproducible”.

        Note also that “bad writing” does not even need to be involved in problems replicating. My own experience during my Master’s provides an excellent example. I used a procedure from the literature to prepare a very specific oxidized tryptophan species. For some reason I did not just get the intended product, but also another one that was not described, and in pretty large amounts. There we go, I was unable to reproduce the literature!

        It took me some time to come up with a possible explanation, which was almost immediately dismissed by my supervisor: I had closed the reaction vessel, and the reaction involved an equilibrium with a gaseous compound, so this may have mattered. I had good reason to close the vessel, because that gaseous compound was dimethyl sulfide, which smells really, really bad in the ‘large’ amounts formed during the reaction. Fortunately I was stubborn enough to test my hypothesis and try with the reaction vessel open. I was now able to reproduce the findings, gone was this unwanted side-product!

        Was it bad writing of the original literature that they did not mention they left their reaction vessel open? If they had not tested with the reaction vessel closed (which is very likely), they would also not have known this mattered. It would be one of those “trivial” things that we don’t think about, but that actually do matter. Like male vs a female animal handler: http://www.nature.com/news/male-researchers-stress-out-rodents-1.15106

  3. The lack of reproducibility is often due to a lack of statistical power. If studies are underpowered then there are two possible problems. If we obtain a significant result then it is likely to be an error, because if there is a real effect then the chance of detecting it is low unless the effect is huge. Now if an effect is observed then we are unlikely to find it in a replication if the effect is real because the experiment is underpowered.

    So the answer is not getting rid of p-values, although thinking about evidence rather than rejection is a good idea. What is important is to run properly powered experiments. That means being honest about meaningful differences and variances. Psychology is among the worst and why they probably have the most problems with replication. Neuroscience is just horrible, use 10 per group but run with 40 outcomes so something will look good, but will never be replicated.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.