The results of any scientific study should be reproducible, right?
One of the main tenets of scientific inquiry is that we should be able to reach the same conclusion over and over again. So, what happens when the results of a study don‘t replicate?
Fewer Than Forty Percent
The answer may seem simple at first. You might think when a study can‘t be reproduced, its conclusions must be false. And they might be.
But there can also be other explanations. When the Reproducibility Project tried to replicate one hundred psychological studies, they succeeded with fewer than 40%. A shockingly low number.
But that doesn't necessarily mean those studies should be thrown out or discredited. Jay Van Bavel, a professor of psychology from NYU, considered how and why this might have occurred, and formed a testable hypothesis of his own. He considered whether the failure to replicate could have been due to differences between the contexts in which the experiments were performed.
Context
Something as simple as an experiment done on mice can be heavily impacted by contextual factors. Consider that everything from the type of food the mice eat, the materials used to line their cages, the temperature that the lab is kept at, and how much light they see can alter their biological processes, such as metabolism.
Just imagine how complex contextual factors might be among people. Professor Van Bavel‘s investigation found that yes, the studies with topics more sensitive to context – which might mean time, location, or demographic differences – were also the least likely to replicate successfully.
Thank you to Jay Van Bavel of New York University for reviewing this episode's script.
Sources And Further Reading:
- Van Bavel, Jay. "Contextual sensitivity in scientific reproducibility." PNAS. Published online 2016 May 23. doi: 10.1073/pnas.1521897113
- Van Bavel, Jay. "Why Do So Many Studies Fail to Replicate?" The New York Times: Gray Matter. May 27, 016. Web. Date of Access: October 9, 2017.
- Vedantam, Shankar. "When Great Minds Think Unalike: Inside Science‘s ‘Replication Crisis‘." Audio blog post. Hidden Brain. NPR, May 24, 2016. Web. Date of Access: October 9, 2017.
