Reproducibility in Science

00:00:11.23 Hi, my name is Shai Silberberg
00:00:14.13 I'm the program director at
00:00:15.28 the National Institute of Neurological Disorders
00:00:17.27 and Stroke at the National Institutes of Health.
00:00:20.05 However, today I'm here to provide my
00:00:22.13 personal perspective on the so-called reproducibility crisis.
00:00:26.16 This is a timely issue,
00:00:29.07 because these days it appears that every other
00:00:31.14 week in either the lay or scientific literature
00:00:34.28 highlighting problems with the ability to
00:00:37.09 reproduce scientific findings.
00:00:39.00 Sadly, many of these publications
00:00:42.09 either implicitly or explicitly
00:00:44.08 imply wrongdoing on the part of scientists.
00:00:47.02 Unjustly tainting the scientific community.
00:00:50.14 So to help put things in perspective,
00:00:53.03 let's describe what it means when results
00:00:55.18 are reproducible, and then identify factors
00:00:59.04 that might lead to low reproducibility.
00:01:01.06 As I'm sure you all know,
00:01:04.05 advances in science are primarily
00:01:06.10 built on prior findings.
00:01:07.26 Results that do not stand the test of time
00:01:10.22 get abandoned, whereas results
00:01:13.00 that can be reproduced serve as
00:01:15.00 foundation for future discoveries.
00:01:17.12 In this process, however, results
00:01:20.01 that cannot be reproduced
00:01:21.21 can slow the progress of science
00:01:23.23 and therefore, it is important to identify
00:01:25.28 the key factors that lead to
00:01:28.09 low reproducibility and to find mechanisms
00:01:31.26 to mitigate them to the best of our ability.
00:01:35.06 Now there are many reasons that can lead to low
00:01:37.19 reproducibility. I list three of them over here.
00:01:41.05 So, for one, complex innovative techniques
00:01:45.01 developed by one group might require
00:01:47.12 extensive training before it can be
00:01:49.25 utilized by others.
00:01:51.05 Until this knowledge is gained
00:01:53.13 it might appear that their initial results aren't
00:01:56.11 reproducible.
00:01:57.11 Second, at times, variables which influence
00:02:02.07 the outcome of the experiment are
00:02:04.07 unknown to the investigator
00:02:05.25 and therefore, they are not recorded.
00:02:08.02 If these so-called "confounding variables"
00:02:11.24 happen to be different in labs trying to reproduce
00:02:15.13 the results, they might not get the same answer.
00:02:18.06 Now an interesting example of a confounding variable
00:02:22.00 is the story of the horse, Clever Hans.
00:02:24.16 This was a horse that was believed to be able
00:02:27.00 to solve arithmetic problems.
00:02:28.17 His trainer would say, "Hans!
00:02:30.06 3 x 2." And the horse would kick
00:02:32.24 six times.
00:02:33.18 It turned out, however,
00:02:35.29 that this very clever horse
00:02:37.18 was actually reading the body language
00:02:39.29 of the trainer, who was totally unaware
00:02:42.22 that he was providing physical cues
00:02:45.03 when the horse should stop kicking.
00:02:47.00 So, in this case, the trainer is
00:02:49.07 the confounding variable.
00:02:50.07 You remove the trainer from the room,
00:02:52.12 and the horse can no longer count.
00:02:54.11 The third issue I mention here,
00:02:56.27 issues with resources.
00:02:59.06 Much is being said in recent years
00:03:02.09 about misidentification of cell lines,
00:03:05.05 and the poor quality of many antibodies
00:03:08.09 and other reagents.
00:03:09.12 If two labs think that they are using the same cell line
00:03:13.07 to do similar experiments, but in actual fact,
00:03:15.24 they are using very different cells.
00:03:17.21 Then it wouldn't be surprising
00:03:19.09 if they don't get the same result.
00:03:21.02 As you can see, none of these three variables
00:03:24.10 that I provide here have anything to do
00:03:26.11 with the integrity of the scientist producing
00:03:29.01 the work.
00:03:30.01 However, we could improve
00:03:32.02 the reproducibility of such experiments
00:03:33.27 if there was greater transparency in the reporting
00:03:37.07 of experimental design, conduct, and analysis.
00:03:40.19 So with that said,
00:03:42.16 I would like to focus my talk on what I personally
00:03:44.29 believe are major contributing factors
00:03:47.11 to low reproducibility.
00:03:48.23 And these are unintentional and unconscious bias,
00:03:52.23 what I call "human nature,"
00:03:54.14 and chance and publication bias.
00:03:57.25 Let's start with human nature.
00:04:00.21 In his amazing Cooper Union Address,
00:04:03.26 Abraham Lincoln said that
00:04:05.22 "human action can be modified
00:04:07.04 to some extent, but human nature
00:04:09.03 cannot be changed."
00:04:10.07 This is a profound statement.
00:04:12.19 Because what he's actually saying
00:04:15.22 is that we have to recognize, as humans,
00:04:18.02 that we have limitations. And we have
00:04:20.01 to do our best to minimize their impact.
00:04:23.05 Now, obviously, Mr. Lincoln was not talking
00:04:26.17 about science. But four decades later
00:04:29.00 Thomas Chamberlin did just that.
00:04:31.03 Thomas Chamberlin was a geologist,
00:04:33.08 and what he proposed was when
00:04:35.06 scientists design experiments,
00:04:36.23 they should test multiple hypotheses
00:04:40.05 in parallel, not just one.
00:04:41.26 Because if we choose our favorite hypothesis,
00:04:44.08 then unconsciously, we tend to magnify results
00:04:48.26 that support our hypothesis
00:04:50.20 and overlook results that do not.
00:04:52.15 So Chamberlin was describing one of many
00:04:56.10 forms of bias, which are inherent
00:04:58.16 to all human beings, including scientists.
00:05:00.20 Importantly, all these biases are unintentional
00:05:05.02 and unconscious. And therefore,
00:05:07.14 have absolutely nothing to do with
00:05:09.00 scientific misconduct.
00:05:10.22 Now bias is very real.
00:05:13.14 It's not a myth, and it lurks behind
00:05:15.23 every corner. To give you just one example,
00:05:18.12 here's a study that was conducted more than
00:05:20.11 half a century ago.
00:05:21.21 And in this study, the investigators
00:05:24.00 used, as you can see, 14 rodents
00:05:26.09 and 39 Homo sapiens in the form of students.
00:05:29.18 Now, this ratio seems rather odd
00:05:33.10 until you realize that the experimental subjects
00:05:35.29 in this experiment were actually the students.
00:05:38.00 This was part of a course in experimental psychology
00:05:42.07 and the students were asked over an entire semester
00:05:45.03 to train animals to do different tasks
00:05:48.03 and they measured how long it took them to learn.
00:05:50.12 Now there was a trick to this experiment
00:05:52.29 the students were lead to believe that there were two
00:05:55.21 types of animals, there were dull animals
00:05:58.11 that were genetically bred to be smart,
00:06:01.02 and then there were the bright animals
00:06:02.26 that were genetically bred to be smarter.
00:06:04.29 Now in actual fact, these were all
00:06:07.02 randomly selected animals.
00:06:08.29 Now before each experiment
00:06:11.04 the students were told whether on that particular
00:06:13.15 day they were going to be studying
00:06:14.24 a dull or bright animal.
00:06:16.21 Now, as I'm sure you've guessed by now
00:06:19.19 at the end of the experiment, at the end of the semester,
00:06:22.02 after repeating this experiment five times
00:06:24.27 it was found, that on average the students found
00:06:27.27 the bright animals learned faster.
00:06:30.18 Now it doesn't take a rocket scientist to figure out
00:06:33.11 what could've been done to minimize
00:06:35.27 this expectation bias.
00:06:38.02 All that was needed was to mask
00:06:39.25 the identity of the animals
00:06:41.13 from the students conducting the experiments
00:06:45.00 and the students analyzing the results.
00:06:47.01 This is known as blinding,
00:06:50.04 and it's an effective way to minimize
00:06:51.27 some types of bias.
00:06:54.07 And in most cases, it is not that difficult to employ
00:06:59.00 or to use blinding.
00:07:00.28 All that is needed, is to ask someone in
00:07:03.18 the lab or the adjacent lab
00:07:05.13 to code the samples and to reveal the code
00:07:07.22 only after the experiments are complete.
00:07:10.03 So blinding should be a serious consideration
00:07:13.17 when conducting experiments,
00:07:15.14 however, if blinding is actually used,
00:07:18.24 it's often not reported.
00:07:20.09 To give you just one example,
00:07:22.12 in this study, the investigators
00:07:24.29 scan seven high profile journals
00:07:27.26 for papers that included animal
00:07:30.16 trials and that were cited at least five hundred times each.
00:07:34.15 These are high impact papers.
00:07:36.13 And what they found, was that while approximately
00:07:40.01 70% of the papers
00:07:41.23 provided information on how the effect depended
00:07:45.23 on the concentration of the compound being used,
00:07:48.23 what's known as a dose response curve,
00:07:50.19 only 20% reported whether the experiments were blinding.
00:07:54.27 And even less reported
00:07:56.17 on whether the animals were randomly assigned
00:07:58.21 to the different comparison groups,
00:08:00.19 a topic I don't have time to discuss now.
00:08:02.26 So, as you can see, there's lack
00:08:07.11 of transparency and reporting of blinding
00:08:09.25 but this is not only a matter of reporting
00:08:12.29 there's indirect evidence that
00:08:14.11 it's also not being used.
00:08:16.13 So to give you one example,
00:08:19.13 I'm giving a very schematic view here
00:08:20.20 in this study, the investigators looked
00:08:23.06 at all the papers that examined the effect
00:08:26.01 of a particular compound in an experimental model
00:08:29.06 of stroke.
00:08:30.05 And what they did, was for each paper,
00:08:32.23 they looked to see how many
00:08:34.18 important parameters were described in the paper
00:08:37.08 that might influence bias,
00:08:38.29 for example, blinding and randomization.
00:08:41.04 And they gave a score for the overall
00:08:44.17 parameters that were reported in the paper.
00:08:47.03 And in parallel, they also recorded
00:08:49.29 at what magnitude effect the drug had
00:08:53.12 in that particular paper.
00:08:54.23 And those results are shown here.
00:08:57.13 What you can see, is that
00:09:00.04 the more parameters reported
00:09:02.17 the smaller the reported effect size
00:09:05.11 or the reverse, the fewer parameters reported
00:09:09.12 the greater the effect size.
00:09:11.04 Suggesting that lack of reporting
00:09:14.07 is associated with lack of practice
00:09:16.16 and that lack of practice can lead to bias.
00:09:19.13 Now not all forms of bias are straightforward
00:09:23.02 like blinding. So, let's imagine
00:09:25.21 -- this is to help you imagine, okay?
00:09:30.24 So, let's imagine,
00:09:32.26 that for the last few weeks
00:09:35.00 you've been on pins and needles
00:09:36.14 waiting to hear if your very first publication
00:09:39.29 will be accepted for publication
00:09:41.04 in a high profile journal.
00:09:43.01 And then finally,
00:09:45.09 you get an answer from the editor.
00:09:47.16 And the answer says that if
00:09:50.13 you can fortify your results with just one more
00:09:53.01 set of experiments, they will seriously
00:09:55.11 consider your manuscript for publication.
00:09:57.20 Oh, you're so excited!
00:09:59.23 Your very first paper is so close.
00:10:02.14 And you're confident that it will take you no more
00:10:05.26 than two weeks to get the desired results.
00:10:07.27 Wait a minute. Hold that thought.
00:10:14.00 Did I just hear you think that
00:10:16.20 you are sure you can get the right answer
00:10:19.00 within a couple of weeks?
00:10:20.21 If I read your thoughts correctly,
00:10:23.04 then you might have just jeopardized your
00:10:25.17 credibility as a scientist.
00:10:27.22 And in fact, you might have even contributed
00:10:30.23 to the reproducibility crisis.
00:10:33.08 If you precede with the experiments
00:10:36.08 without carefully designing them
00:10:38.13 to minimize bias, such as blinding yourself
00:10:41.06 then expectation bias
00:10:43.23 and the pressures to publish high profile papers
00:10:47.08 might lead you to get the desired result
00:10:51.08 instead of the true result.
00:10:52.26 So, as you can see, it's extremely important
00:10:56.20 to design your experiments
00:10:58.12 to minimize bias.
00:10:59.27 With that said, I'd like to switch to the second topic
00:11:03.08 I mentioned earlier,
00:11:04.16 and that is publication bias.
00:11:06.05 As we all know, we typically start
00:11:09.17 research with a question,
00:11:11.16 we then generate a hypothesis to test it.
00:11:14.06 And then design experiments to test that hypothesis.
00:11:17.06 Now, we do the experiments
00:11:19.29 and if the results support our hypothesis,
00:11:22.15 we're often very happen, and also very proud of ourselves
00:11:25.10 for coming up with such a great hypothesis
00:11:27.20 and we'll do our best to publish our results.
00:11:30.07 These are all positive results.
00:11:32.17 If, however, we get what are called negative results,
00:11:35.27 the results do not support our hypothesis,
00:11:37.22 many times, we'll simply file them away
00:11:41.27 and they won't see the light of day.
00:11:43.17 This is also known as the file drawer problem.
00:11:46.19 So if you now combine publication bias
00:11:49.29 with the element of chance,
00:11:51.18 then you have a recipe for low reproducibility.
00:11:55.01 To help understand this point,
00:11:57.18 let's look at a really cool study
00:11:59.19 conducted by the ALS therapeutics institute
00:12:03.01 and published back in 2008.
00:12:04.26 What they did is, they had information about
00:12:08.27 this animal model called the SOD1 mouse,
00:12:12.21 it's an animal model used to study ALS.
00:12:15.17 And they had information about many, many animals
00:12:18.29 that served as controls
00:12:20.15 in different experiments.
00:12:21.25 In fact, they had more than 2000 animals.
00:12:23.28 And they knew everything about these animals,
00:12:26.13 when they were born and when they died,
00:12:28.01 and what litter they came from,
00:12:29.13 what their sex was, and so on and so forth.
00:12:31.14 And so what they did was, they put all this information
00:12:34.26 into a database, and then they conducted this really cool
00:12:38.24 in silico experiment. What they did is
00:12:42.05 they randomly selected animals from this database
00:12:44.29 and put them in two separate groups.
00:12:47.02 And they asked, what is the probability
00:12:50.15 of seeing at least a 5% difference
00:12:53.04 in the mean life expectancy
00:12:55.03 between these two groups.
00:12:56.17 Which, by definition would be
00:12:58.01 by chance because all these
00:12:59.12 animals are control animals
00:13:00.25 that lived out their lives.
00:13:02.12 And what they found after repeating the experiment
00:13:06.14 thousands and thousands of times,
00:13:07.26 that if you have ten animals per group,
00:13:11.08 then on average, you have about
00:13:15.11 10% chance of seeing significant result
00:13:18.11 of 5% difference in their life expectancy.
00:13:22.06 Now some of you, might think that getting the right answer
00:13:26.23 9 out of 10 times is not bad,
00:13:28.13 but in actual fact, if you combine this
00:13:31.04 with publication bias,
00:13:34.11 you can see how this is a problem.
00:13:36.21 So, let's imagine,
00:13:38.26 that there are ten groups that decide to test
00:13:41.29 an effect of a particular drug
00:13:43.18 in this ALS model.
00:13:45.00 Nine groups see that the drug has no effect
00:13:48.21 and they file the results away, they're not interesting
00:13:51.16 and they don't publish them.
00:13:52.13 And then one group by chance,
00:13:54.25 gets an effect and this is what gets published in a high profile journal.
00:13:59.17 So you can see how publication bias
00:14:02.22 and chance gave rise to an irreproducible result.
00:14:05.29 So what can we do, in order to improve reproducibility?
00:14:10.29 Well, first and foremost,
00:14:13.10 we need to increase transparency in reporting,
00:14:17.02 and to do that, we need to use review.
00:14:20.09 We need reviewers of manuscripts and grants
00:14:23.19 to pay greater attention to the information
00:14:26.13 provided in these documents.
00:14:28.08 The importance of transparency in reporting
00:14:32.04 has been recently recognized by both journals
00:14:34.15 and funding agencies.
00:14:36.00 So, for example, NIH
00:14:38.10 has recently made changes to their
00:14:40.00 review process to assure that particular important
00:14:43.11 information is included in the grant applications.
00:14:45.08 Several journals have added checklists
00:14:49.01 to help them identify which parameters are included
00:14:53.24 in the manuscript and which are not.
00:14:55.21 The second point, is that we need to somehow deal
00:15:00.00 with unconscious bias and deficiencies in experimental
00:15:03.16 procedures. To this, what we need
00:15:06.29 is education. We need to train the scientific
00:15:10.01 community to understand that
00:15:13.01 we are all prone to bias
00:15:15.10 and that therefore, precautions that we take during
00:15:18.00 the experiment are not just for the sake of
00:15:20.28 adding difficulty, but they actually serve an important purpose.
00:15:25.12 Dealing with publication bias is a greater
00:15:29.04 nut to crack, because that will require change in culture.
00:15:32.12 We need to somehow shift the focus from glitter
00:15:35.11 and focus more on rigor.
00:15:37.29 So, if you're part of the scientific enterprise,
00:15:43.02 I would like to point you to a few important
00:15:46.22 points. First, we must all recognize that we
00:15:50.05 are all prone to unconscious bias.
00:15:52.21 And because this is the case,
00:15:55.04 we need to critically assess results and publications,
00:15:58.13 we need to rigorously design, execute,
00:16:03.07 and analyze experiments.
00:16:04.24 We should plan experiments to disprove
00:16:08.12 the hypothesis, not to prove that it's right.
00:16:10.13 And we should need to favor robust findings,
00:16:15.08 but if it appears to be too good to be true,
00:16:17.13 it probably is.
00:16:20.06 So I'd like to end with the words
00:16:22.03 of the great physicist, Richard Feynman.
00:16:26.06 Who said that if you're doing an experiment,
00:16:28.14 you should report everything that you think
00:16:31.00 might make it invalid,
00:16:32.19 not only what you think is right about it.
00:16:35.02 Think about these profound words
00:16:37.25 the next time you're reading,
00:16:40.07 writing, or reviewing a manuscript.
00:16:43.06 I'd like to thank you for your attention
00:16:46.17 to thank NINDS leadership
00:16:49.15 for giving me the opportunity to devote time
00:16:52.00 to this important topic,
00:16:53.19 Andrew Nowak for photography,
00:16:56.10 Lauren Ullrich for illustrations,
00:16:59.07 and my daughter, Leeron, for putting a smile on my face.
00:17:02.17 Thank you.

This Talk

Speaker: Shai Silberberg

Audience:

General Public
Educators of H. School / Intro Undergrad
Educators of Adv. Undergrad / Grad
Researcher
Educators

Recorded: April 2016

Talk Overview

In the scientific community, problems may arise when different labs cannot reproduce certain experiments. In this lecture, Dr. Silberberg focuses on two of the components that contribute to low reproducibility: unconscious bias (human nature) and publication bias (a tendency to publish only results that support our hypothesis). Silberberg suggests that the incorporation of randomization and blind experiments, and more transparent reporting will increase the quality of both experiments and publications. If these metrics are enforced by the grant and journal review processes, reproducibility in science will improve.

Dr. Shai Silberberg is a Program Director at the National Institute of Neurological Disorders and Stroke (NINDS) in the National Institutes of Health (NIH) where he leads the Institute’s efforts to increase the excellence of science and the completeness of research reporting. Dr. Silberberg obtained a Ph.D. in Neurophysiology from the Hebrew University in Jerusalem. Prior to joining NINDS, Dr. Silberberg was an Associate Professor at Ben-Gurion University of the Negev in Israel, investigating the biophysical functions and physiological roles of various ion channels.

Speaker Bio

Shai Silberberg

Shai Silberberg

Dr. Shai Silberberg is a Program Director at the National Institute of Neurological Disorders and Stroke (NINDS) in the National Institutes of Health (NIH) where he leads the Institute’s efforts to increase the excellence of science and the completeness of research reporting. Dr. Silberberg obtained a Ph.D. in Neurophysiology from the Hebrew University in Jerusalem…. Continue Reading