Y: The results of any scientific study should be reproducible, right? One of the main tenets of scientific inquiry is that we should be able to reach the same conclusion over and over again. So what happens when the results of a study don’t replicate?
D: The answer may seem simple at first. You might think that when a study can’t be reproduced, its conclusions must be false. And they might be. But there can also be other explanations.
Y: When the Reproducibility Project tried to replicate one hundred psychological studies, they succeeded with fewer than 40%. A shockingly low number. Does that mean that most of the original studies should now be thrown out or discredited?
D: Not necessarily. Jay Van Bavel, a professor of psychology from NYU, considered how and why this might have occurred, and formed a testable hypothesis of his own. He considered whether the failure to replicate could have been due to differences between the contexts in which the experiments were performed.
Y: Something as simple as an experiment done on mice can be heavily impacted by contextual factors. Consider that everything from the type of food the mice eat, the materials used to line their cages, and how much light they see can alter their biological processes, such as metabolism. Just imagine how complex contextual factors might be among people.
D: Professor Van Bavel’s investigation found that yes, the studies with topics that were more sensitive to context – which might mean time, location, or demographic differences – were also the least likely to replicate successfully.
