False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
⚡ Contested ↗📌 Appears in:
Plain English Summary
One of the most important — and funniest — papers in modern science. Published alongside a famous study claiming to prove psychic powers, it asked: how easy is it to get a significant result when nothing real is happening? Terrifyingly easy. Simulating 15,000 samples, the authors showed common research shortcuts — choosing which measurements to report, peeking at data mid-collection, picking control variables — inflate false-positive rates from 5% to between 7.7% and 12.6% individually. Combine all four and you hit a jaw-dropping 61%. To prove it, they produced "significant" evidence a Beatles song makes people literally younger. The paper helped launch the pre-registration movement and became a blueprint for transparent research.
Abstract
In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
Links
Related Papers
Companion
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J.E (2014)
- Commentary: Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? — Heino, Matti T. J (2017)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice — LeBel, Etienne P (2011)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect — Ritchie, Stuart J (2012)
Cited By
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? — Pashler, Harold (2012)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory (2012)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling — John, Leslie K (2012)
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E (2024)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology — Bierman, Dick J (2016)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
- Decline Effects: Types, Mechanisms, and Personal Reflections — Protzko, John (2017)
- Religious Priming: A Meta-Analysis With a Focus on Prosociality — Shariff, Azim F (2015)
- Why Science Is Not Necessarily Self-Correcting — Ioannidis, John P.A (2012)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
Also by these authors
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Simmons, Joseph P, Nelson, Leif D, Simonsohn, Uri (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science. https://doi.org/10.1177/0956797611417632
@article{simmons_2011_false_positive,
title = {False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant},
author = {Simmons, Joseph P and Nelson, Leif D and Simonsohn, Uri},
year = {2011},
journal = {Psychological Science},
doi = {10.1177/0956797611417632},
}