Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi
β‘ Contested βπ Appears in:
Plain English Summary
When a well-known psychologist published data apparently showing people can see the future, Wagenmakers and colleagues said: not so fast. They re-crunched all nine of Daryl Bem's precognition experiments using Bayesian statistics -- a method that directly measures how much evidence supports one idea over another. The verdict was brutal: only one out of ten tests showed even modest support for psychic powers, while three actually favored the boring explanation that nothing paranormal happened. The rest were a statistical shrug. The team pinpointed where Bem went wrong: mixing up exploratory fishing expeditions with rigorous hypothesis testing, and leaning on p-values that dramatically oversell the evidence. They capped it off with a practical reform blueprint, including pre-registering studies and inviting skeptics to collaborate on experiments. This paper became a rallying cry in psychology's broader reckoning with its own statistical habits.
Abstract
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively aο¬ect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory, and that one-sided p-values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data using a default Bayesian t-test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly conο¬rmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p-values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Links
Related Papers
Companion
- A Bayes Factor Meta-Analysis of Bem's ESP Claim β Rouder, Jeffrey N (2011)
- Back from the Future: Parapsychology and the Bem Affair β Alcock, James E (2011)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice β LeBel, Etienne P (2011)
- Correcting the Past: Failures to Replicate Psi β Galak, Jeff (2012)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant β Simmons, Joseph P (2011)
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? β Pashler, Harold (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling β John, Leslie K (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time β Gelman, Andrew (2013)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect β Ritchie, Stuart J (2012)
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? β Wagenmakers, Eric-Jan (2012)
- Mindless Statistics β Gigerenzer, Gerd (2004)
- Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology β Meehl, Paul E (1978)
Cites
- Meta-Analysis of Free-Response Studies, 1992β2008: Assessing the Noise Reduction Model in Parapsychology β Storm, Lance (2010)
- Replication and Meta-Analysis in Parapsychology β Utts, Jessica (1991)
- Why Is Psi So Elusive? A Review and Proposed Model β Kennedy, James E (2001)
- Why Most Published Research Findings Are False β Ioannidis, John P.A (2005)
Cited By
- Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018) β Houran, James (2018)
- Bem's 'Feeling the Future' (2011) Five Years Later: Its Impact on Scientific Literature β Silva, Bruno A (2017)
- Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus β Rabeyron, Thomas (2020)
- Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses β Lakens, DaniΓ«l (2017)
- The Experimental Evidence for Parapsychological Phenomena: A Review β CardeΓ±a, Etzel (2018)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time β Gelman, Andrew (2013)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research β Kennedy, J.E (2014)
- Replication Unreliability in Psychology: Elusive Phenomena or "Elusive" Statistical Power? β Tressoldi, Patrizio E (2012)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability β Nosek, Brian A (2012)
Also by these authors
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
Paranormal psychic believers and skeptics: a large-scale test of the cognitive differences hypothesis
π Cite this paper
Wagenmakers, Eric-Jan, Wetzels, Ruud, Borsboom, Denny, van der Maas, Han (2011). Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0022790
@article{wagenmakers_2011_why_psychologists,
title = {Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi},
author = {Wagenmakers, Eric-Jan and Wetzels, Ruud and Borsboom, Denny and van der Maas, Han},
year = {2011},
journal = {Journal of Personality and Social Psychology},
doi = {10.1037/a0022790},
}