The "File Drawer Problem" and Tolerance for Null Results
📄 Original study ↗Plain English Summary
Imagine scientists only publishing their exciting findings while stuffing boring "nothing happened" results into a file drawer. That's the "file drawer problem," and this landmark paper tackled it head-on. Rosenthal invented a clever tool called the "fail-safe N" — a formula that calculates how many hidden, unpublished null studies would need to exist in those file drawers to wipe out a published positive finding. The numbers can be staggering: in one example, you'd need over 3,200 buried studies to overturn 94 published ones, and nearly 50,000 to cancel out 311. He proposed a handy rule of thumb for when results are sturdy enough to trust despite possible hidden studies. This formula became absolutely essential in parapsychology research, where every meta-analysis now uses it to argue whether cumulative evidence for psychic phenomena can survive the file drawer threat.
Abstract
For any given research area, one cannot tell how many studies have been conducted but never reported. The extreme view of the "file drawer problem" is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show nonsignificant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
Links
Related Papers
Companion
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory (2012)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- Scientists behaving badly — Martinson, Brian C (2005)
Cited By
- Evidence for Consciousness-Related Anomalies in Random Physical Systems — Radin, Dean I (1989)
- Does Psi Exist? Replicable Evidence for an Anomalous Process of Information Transfer — Bem, Daryl J (1994)
- "Future Telling": A Meta-Analysis of Forced-Choice Precognition Experiments, 1935-1987 — Honorton, Charles (1989)
- Examining Psychokinesis: The Interaction of Human Intention With Random Number Generators—A Meta-Analysis — Bösch, Holger (2006)
- Why Science Is Not Necessarily Self-Correcting — Ioannidis, John P.A (2012)
- Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology — Bierman, Dick J (2016)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Rosenthal, Robert (1979). The "File Drawer Problem" and Tolerance for Null Results. Psychological Bulletin. https://doi.org/10.1037/0033-2909.86.3.638
@article{rosenthal_1979_file_drawer,
title = {The "File Drawer Problem" and Tolerance for Null Results},
author = {Rosenthal, Robert},
year = {1979},
journal = {Psychological Bulletin},
doi = {10.1037/0033-2909.86.3.638},
}