Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology
⚡ Contested ↗📌 Appears in:
Plain English Summary
Clever statistical detective work targeting Daryl Bem's famous 2011 precognition experiments. Francis asks a simple question: Bem reported positive results in 9 out of 10 studies, but given how small the effects were, should that many have actually worked? The expected hit rate was only about 6 out of 10 — meaning Bem's near-perfect record is itself suspiciously unlikely. Francis finds the same too-good-to-be-true pattern in unrelated verbal overshadowing research. The diagnosis: publication bias — the tendency to publish wins and bury losses — likely contaminated both literatures, making them unreliable as evidence. He suggests Bayesian analysis as a healthier alternative.
Actual Paper Abstract
Empirical replication has long been considered the final arbiter of phenomena in science, but replication is undermined when there is evidence for publication bias. Evidence for publication bias in a set of experiments can be found when the observed number of rejections of the null hypothesis exceeds the expected number of rejections. Application of this test reveals evidence of publication bias in two prominent investigations from experimental psychology that have purported to reveal evidence of extrasensory perception and to indicate severe limitations of the scientific method. The presence of publication bias suggests that those investigations cannot be taken as proper scientific studies of such phenomena, because critical data are not available to the field. Publication bias could partly be avoided if experimental psychologists started using Bayesian data analysis techniques.
Research Notes
Key skeptical contribution to the Bem Feeling the Future controversy. Demonstrates that the seemingly impressive replication rate across Bem's experiments is itself statistically implausible, providing a quantitative basis for the suspicion that selective reporting inflated the evidence for precognition.
Applying the Ioannidis and Trikalinos (2007) test for excess significance to Bem's (2011) ten psi experiments and a set of verbal overshadowing studies, this analysis finds that the observed number of null hypothesis rejections substantially exceeds what would be expected given the experiments' statistical power. Bem's studies yield a pooled effect size of g* = 0.186, predicting 6.27 rejections out of 10, yet 9 were reported (p = .058). The verbal overshadowing literature shows a similar pattern (p = .022). These results indicate publication bias contaminates both literatures, rendering them uninformative as scientific evidence. Bayesian data analysis is proposed as a partial remedy.
Links
Related Papers
Cites
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Must Psychologists Change the Way They Analyze Their Data? — Bem, Daryl J (2011)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim — Rouder, Jeffrey N (2011)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- A Practical Solution to the Pervasive Problems of p Values — Wagenmakers, Eric-Jan (2007)
Companion
- Correcting the Past: Failures to Replicate Psi — Galak, Jeff (2012)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice — LeBel, Etienne P (2011)
- The "File Drawer Problem" and Tolerance for Null Results — Rosenthal, Robert (1979)
- Small Telescopes: Detectability and the Evaluation of Replication Results — Simonsohn, Uri (2015)
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Francis, Gregory (2012). Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-012-0227-9
@article{francis_2012_publication_bias,
title = {Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology},
author = {Francis, Gregory},
year = {2012},
journal = {Psychonomic Bulletin & Review},
doi = {10.3758/s13423-012-0227-9},
}