Do interventions to reduce misconduct actually work? Maybe not, says new report

Can we teach good behavior in the lab? That’s the premise behind a number of interventions aimed at improving research integrity, invested in by universities across the world and even private companies. Trouble is, a new review from the Cochrane Library shows that there is little good evidence to show these interventions work. We spoke with authors Elizabeth Wager (on the board of directors of our parent organization) and Ana Marusic, at the University of Split School of Medicine in Croatia.

Retraction Watch: Let’s start by talking about what you found – looking at 31 studies (including 15 randomized controlled trials) that included more than 9500 participants, you saw there was some evidence that training in research integrity had some effects on participants’ attitudes, but “minimal (or short-lived) effects on their knowledge.” Can you talk more about that, including why the interventions had little impact on knowledge?

Elizabeth Wager and Ana Marusic: Because studies use different measures of success, we grouped them into those looking at effects on behavior, attitude or knowledge. Obviously, it’s easier to measure researchers’ attitudes towards misconduct than to measure actual misconduct. You would expect training should increase knowledge, but many studies showed little effect. This might be because most researchers already know about types of misconduct and it isn’t lack of knowledge that causes them to go astray. A few studies looked at effects immediately after training but found these had worn off after a few months.

RW: You also said that evidence was very “low quality” – what does that mean? And does it mean we should not place much emphasis on the findings of this review, since it’s based on somewhat unreliable evidence?

EW and AM: Many of the published studies weren’t well designed to reduce bias and we excluded many that didn’t have proper control groups. Of those we did include, many didn’t randomize participants or blind raters to the intervention groups. We also found many studies used unvalidated scoring systems, especially those trying to measure attitudes or knowledge.

We were also struck by how little detail was given in the descriptions of the interventions. For example, some of the studies on the effects of running courses on research integrity simply stated the number of sessions and how long they lasted and gave no information about the curriculum, how it was taught, or how teaching quality was checked. So even if the studies had shown a clear effect, it would have been impossible to repeat the experiment or to adopt the teaching methods in other centers.

RW: The one reasonably bright spot appeared to be with plagiarism, where interventions had some effect. Can you talk more about what appeared to work?

EW and AM: Yes, training on plagiarism did seem to be more effective than general training on research integrity although not all the studies showed positive effects. We found several studies that used text-matching software (such as Turnitin) on students’ work, or included practical exercises in paraphrasing text, and some of these showed effects both in students’ understanding of plagiarism and in reducing plagiarism in coursework. We can think of a couple of reasons why plagiarism might respond better to training than other types of misconduct. The first is that definitions of plagiarism are quite technical, so students and researchers may be unaware of them. (In our training, we often come across students who don’t realise that citing a reference doesn’t mean you can copy large chunks from it, so this clearly isn’t instinctive and needs to be learned.) The other reason may be that text-matching tools make plagiarism easy to quantify, so it’s much easier to measure a small change in such behavior than, say, a change in attitude towards guest authorship, which is more nebulous.

RW: Were you disappointed at all by the conclusions of your review?

EW and AM: Yes, I guess we were! It would have been great to find some simple interventions and be able to say to institutions “If you just do this, you will reduce research misconduct.” However, if it really were that easy to reduce misconduct, then probably it would have happened already. Also, we realize that research integrity is a complex phenomenon, likely to be affected by subtle factors in the research environment and incentive systems. So, although we were a bit disappointed, we weren’t all that surprised by our findings.

RW: One issue you note is that the types of interventions varied widely – from online lectures to practical exercises – and the studies that tested them varied, as well, considering different outcome measures. How can we better standardize this field?

EW and AM: That’s a good question. We purposely used a very broad search and didn’t confine ourselves to one discipline (such as medicine) because we hoped that techniques developed in one field might be applicable in others. We were hoping that researchers in psychology, education, social science, or even criminology might have applied their expertise to this topic and come up with creative solutions. Given that none of the techniques seemed particularly effective, I think we need more diversity rather than standardization.

But when it comes to measuring outcomes then we do feel more standardization is needed.  There are very few validated scales (the only one we saw being used a few times was the Plagiarism Knowledge Survey). As we said before, measuring effects on the actual incidence of misconduct is tough, so we need to be realistic about outcome measures, but also ensure they are meaningful. Asking students if they enjoyed a course tells you almost nothing about its real effects, but sadly that’s the most common form of feedback for a lot of training.

RW: You note that the studies were plagued by bias – even in the randomized controlled trials, which are designed to avoid such pitfalls. What kinds of biases did you identify, and how could we avoid them in future trials?

EW and AM: The problem we had with evaluating randomized controlled trials was that the articles did not report important methodological issues, such as the randomization process, whether and how allocation to trial groups was concealed from the participants, or how participants and researchers were blinded. That’s why our assessment of biases was “unclear” in nearly all cases. Maybe the trials were conducted adequately, but they were not reported clearly enough so that the methodological quality could be assessed. The articles were often not from the biomedical field and were published long ago, so it was not possible to contact the authors. How to avoid such problems in future? By adequately reporting the studies, for example following relevant reporting guidelines, such as CONSORT for randomized controlled trials.

RW: There’s a lot of money being invested in these interventions – such as courses being offered by private companies and universities. Do you fear some (or all) of that money is being wasted on interventions that don’t work?

EW and AM:  Frankly, yes. Some training on Responsible Conduct of Research (or RCR) seems to have become a bit of a “tick box” exercise and, when such courses become compulsory, there’s an obvious market for providers to step in. Possibly, commercial providers have unpublished evidence that their systems are effective (which they don’t publish for commercial reasons), but we’d encourage much more transparency and discussion about this.

We’d also like to see investigation of approaches other than training. For example, one of the studies we included looked at the effects of a journal using different forms to collect authorship information. The design of the form had an impact on the truthfulness of the authorship declaration. So there may be simple, low-cost changes at all stages in the research process which might, together, have more effect on reducing misconduct than compulsory training.

RW: Besides these interventions, what else can institutions do to help reduce misconduct?

EW and AM: One reason we know so little about effects of interventions designed to reduce misconduct is that we don’t have reliable information about the frequency of misconduct. Sadly, in most countries, information about investigations at institutions is kept secret, so we have no way of telling what’s happening. We have anecdotal evidence that institutions don’t always investigate suspected misconduct properly, but, even when they do, the results are all too often buried. As well as investment into further research on factors contributing to research integrity, we’d like to see greater transparency.

Lastly, our results indicate that perhaps the research community doesn’t adequately address responsible conduct of research and research misconduct. Maybe we need more qualitative studies to understand what responsible research really is, and how it can be fostered, and what leads to research misconduct. When we understand these phenomena, maybe we will then be more successful in developing and testing interventions to promote good behavior and prevent bad behavior in science!

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.