Lesson one: How to design a good experiment

It’s clear science has a reproducibility problem. What’s less clear is how to address it. Recently, the U.S. National Institutes of Health awarded the Global Biological Standards Institute (GBSI) $2.34 million to train students in good experimental design (also covered by The Scientist). The first week begins June 2018 at Harvard Medical School. We spoke with Leonard P. Freedman, Founding President and Chief Scientific Officer of GBSI and Vivian Siegel, the education director of the program.

Retraction Watch: Why do students need extra training in experimental design?

Len Freedman: We’ve done a number of studies, including a self-published paper, “The Case for Standards in Life Science Research: Seizing Opportunities at a Time of Critical Need,” that deconstructed all the various drivers that contribute to irreproducibility. Some readers may remember our famous — or infamous — economic impact study that estimated that scientists spend $28 billion per year on non-reproducible research. Besides that explosive number, the paper also tried to identify the principle drivers that contribute to irreproducibility, which included misuse or failure to validate reagents, problems with study design, and data analysis and reporting.

GBSI has done a lot of work raising awareness, as well as providing tangible solutions to challenges in the reagent space, such as the problem of misidentified cell lines, and poorly validated research antibodies. Another area we thought we should put effort into that can make an impact down the road is a reassessment of how we train students and postdocs – with a particular focus on experimental design.

Vivian Siegel: So I started working with GBSI to develop education and training to improve research quality. Len and I discussed how we were trained ourselves as experimental biologists; we both recognized that the way we learned to do experiments was by our mentors. The vast majority of scientists rely on this informal training that happens in the lab. And there are few places where scientists can be formally trained on how to do an experiment properly.

RW: Can you briefly explain the “reverse engineering” approach, and what it’s designed to accomplish?

VS: Some places have started teaching experimental design, but they typically begin with philosophical principles, which students passively learn, then move forward on how to apply those principles in research. The problem with that is it’s very difficult to apply learning to your own work. We’ve heard from people teaching these courses, and they’ve told us they spend at least half their time one-on-one with students, trying to help them with this difficult task. In our course, students start with a dataset from an already completed experiment, and work backwards to determine how to design an experiment to produce high quality data, and what analyses they should apply. They learn the philosophical principles experientially, while working with data. For example, students might have a dataset obtained through quantitative microscopy, and discuss how to use the tool to obtain those data; if a particular error occurred, how would it affect the data?

RW: Of course, you don’t want students to think they can start with data and work backwards to design an experiment as usual practice, which can also lead to problems. So how do you make sure they don’t follow this pattern for the rest of their careers?

VS: Of course. The simplest way is to give them examples of the flaws of trying to apply questions to existing data, or change questions so that get “significant results.” You can use the reverse engineering approach to illustrate the interpretive flaws that happen when you create an experiment based on data that already exist.

But we are also in a world when people generate lots of data without a hypothesis — in genomics, for example. In some sense, we always have been — people often become scientists because they have a curiosity about what they see around them, i.e. data. For instance, they notice the days get shorter when it’s colder outside — I wonder if it’s something to do with the earth’s distance from the sun? Let’s design an experiment to find out.

RW: How will the program help boost mentorship capacity?

VS: People that thrive in their professional careers have learned to seek out, recruit, and work with mentors throughout their lives, not just people they work with in the lab. There are a lot of disparities, quite frankly, based on race, socioeconomic status, and other factors. Women and people of color have a harder time building a good network of mentors. This is one of the reasons why this program is funded by NIH, which cares about building a diverse workforce.

There are a few ways mentoring will happen during the program. We’re still refining our ideas. In general, the students will be working with faculty mentors all week, which will provide them with a lifelong network of experts to consult going forward in their careers.

The middle of the week is all about students taking what they learned in the first couple of days and applying it with problem sets or designing experiments. During that day, faculty will be there, along with other researchers experienced in that methodology, to work with them. There will be highly skilled faculty and experienced researcher mentoring happening right there, about this kind of experiment.

It’s not just the students who will get mentoring — the program is designed to build mentoring of faculty, as well. We’re trying to help faculty teach experimental design at their own institutions. We’ll have launch faculty who will teach the first summer, but we’re also inviting faculty who want to do the training themselves who can observe the launch faculty. If we videotape the week, some faculty might not even need to attend. That’s an unusual aspect of project — we have a kind of “train the trainer” component.

RW: A recent review found that there’s little good evidence to show interventions to reduce research misconduct have an impact. How will this intervention be different?

VS: This is such an important question. My first conversation with Len when we decided to work on training to improve research quality touched on this — I live in mortal fear of creating a Responsible Conduct in Research course where students just check a box saying they attended classes about a high-profile problem, but didn’t really learn anything. It’s so hard to change practice. For instance: We know the best practices for authenticating cell lines, they aren’t prohibitively expensive, and yet few people do it.

The key part of our program is that we get students to practice what they’re learning right away. Only the first part of the week involves some passive learning — by the end of the second day, they have to talk about the key points that affect experiments, and by day three, they’re starting to apply what they know. I think that’s going to make a huge difference in what actually goes home with them. Because it’s already gone home with them, in a way.

LF: I strongly agree. It’s one thing to raise students’ awareness about a problem — such as research misconduct — and it’s another thing to actually train them about something, get their hands dirty. That can have a significant impact going forward, as they move into their own labs and their own careers. I’ve spent five years working on reproducibility, and education and training is where we can have a meaningful impact.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here. If you have comments or feedback, you can reach us at [email protected].