The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time
β‘ Contestedπ Appears in:
Plain English Summary
Imagine you're walking through a garden where the path keeps branching, and at each fork you make a choice that feels obvious β but a different person might have chosen differently at every turn. That's the powerful metaphor this paper introduced to explain how researchers can unknowingly inflate their results. Even without deliberately cheating or fishing for significance, the sheer number of small analytical decisions (how to split groups, which outliers to drop, which measure to emphasize) means the one analysis a scientist reports isn't really just "one" test β it's the survivor of many invisible alternatives. Using Bem's precognition studies as a prime example, the authors argue that pre-registration β publicly committing to your analysis plan before seeing data β is the best antidote to this hidden flexibility.
Abstract
Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data. The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of ο¬shing or examining multiple p-values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre-speciο¬ed and, as a result, were contingent on data.
Related Papers
Cites
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant β Simmons, Joseph P (2011)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi β Wagenmakers, Eric-Jan (2011)
- Must Psychologists Change the Way They Analyze Their Data? β Bem, Daryl J (2011)
- Power failure: why small sample size undermines the reliability of neuroscience β Button, Katherine S (2013)
Companion
- Estimating the Reproducibility of Psychological Science β Open Science Collaboration (2015)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim β Rouder, Jeffrey N (2011)
- Registered Reports: A Method to Increase the Credibility of Published Results β Nosek, Brian A (2014)
- An Agenda for Purely Confirmatory Research β Wagenmakers, Eric-Jan (2012)
- Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus β Rabeyron, Thomas (2020)
- Why Most Published Research Findings Are False β Ioannidis, John P.A (2005)
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
π Cite this paper
Gelman, Andrew, Loken, Eric (2013). The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time. Columbia University Department of Statistics Working Paper.
@article{gelman_2013_forking_paths,
title = {The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time},
author = {Gelman, Andrew and Loken, Eric},
year = {2013},
journal = {Columbia University Department of Statistics Working Paper},
}