How many scientists admit to questionable research practices?

Sure, everyone knows it’s not a good idea to falsify data. But what about somewhat lesser offenses that also undermine the reproducibility of your findings, such as only publishing studies that “work,” and reporting an unexpected finding as something you had predicted from the beginning? In 2012, a survey of more than 2,000 psychologists based in the U.S. found that most admitted to adopting at least one “questionable research practice.” But would psychologists in other countries say the same? (Answer: Yes.) A group of researchers led by Franca Agnoli at the University of Padova posed this question to 277 Italian psychologists; their results appear in PLOS ONE. 

Retraction Watch: Why is it important to compare how many researchers engage in questionable practices in different countries? 

Franca Agnoli: In my view, the use of questionable research practices is a serious problem that undermines our science. We need to understand why it is happening and how to prevent it. If we had found substantial differences between countries in use of QRPs, then we would have wanted to explore why the practices are less frequent in some countries and use this knowledge to reduce its frequency throughout the world. Finding, instead, that the differences between countries is small, suggests that the reasons for these practices are found in systemic properties of the international research community.

RW: Were you surprised to see that almost 9 out of 10 Italian psychologists admitted to engaging in at least one questionable practice? 

FA: It is a surprise that researchers from any country are willing to admit in a survey that they adopt practices that may be questionable. Apparently, they believed and trusted in our promises that anonymity would be maintained. It is, of course, disappointing that such a large percentage of Italian researchers has engaged in at least one QRP, but it is no more surprising than the similar results for researchers in the US and Germany.

RW: The Italian researchers engaged in the same rate of questionable practices as US respondents in a previous study. But where they tended to differ was in how many researchers agreed the practice was questionable. In the US study, many researchers who said they engaged in a questionable practice said they thought it was okay — ie, “defensible.” The rate of that among Italian respondents was much lower. So in essence, Italian researchers are engaging in the practice even though they believe it’s often not okay. What is your reaction to those findings?

FA: I am puzzled by the finding that Italian researchers who adopted QRPs were less likely to say that the QRP was defensible. To me, the surprising result is that US researchers who adopted a QRP were very likely to consider its use defensible. Possibly Italian and US researchers interpreted the question somewhat differently due to subtle differences in the languages. For example, the US researchers may have interpreted the question as a reference only to their own use of the QRP, whereas Italian researchers may have interpreted the question more broadly. We do not know the reason for this difference.

RW: You asked researchers about 10 questionable practices. Some were obviously problematic — such as falsifying data, which only few people admitted to. Given that psychology, like other fields, is facing a problem over reproducibility, which other practices do you believe are most problematic, but may not be seen as such?

FA: In response to a question asking for comments about this survey, some participants explained why they thought some of these practices were reasonable and justifiable. They wrote that they considered it appropriate to collect more data if the results weren’t significant or to stop early if the results were significant. These two practices also had the largest percentages of “possibly defensible” responses. If researchers believe that these two QRPs are defensible, then they are likely to use them, with harmful consequences to our science.

RW: Since researchers had the option of not answering the questions, do you wonder if the answers are representative of the community? Presumably, people who engage in these practices regularly might not be willing to answer questions about it — or at least, not answer them honestly. 

FA: Yes, we have to wonder how our survey respondents differ from non-respondents. We simply don’t know how respondents and non-respondents differ. We cannot assume that people who regularly engage in these practices are less likely to respond, or respond honestly.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.