Why Science Is Not Necessarily Self-Correcting
📄 Original study📌 Appears in:
Plain English Summary
We like to think science fixes its own mistakes, but does it really? Ioannidis digs into the numbers and the picture is grim: only about 1% of psychology papers even try to repeat earlier work, and a tiny fraction of those are truly independent do-overs. With most studies running on weak statistical power (around 35%), somewhere between 30% and 95% of published 'significant' findings could simply be wrong and nobody's checking. He identifies thirteen roadblocks to self-correction, from journals refusing to publish replication attempts to researchers cherry-picking their best-looking results. This paper became a touchstone for both critics and supporters of parapsychology research, since both sides point to these same problems to argue their case. Fixes exist, but each comes with trade-offs.
Actual Paper Abstract
The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large. History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that "it is obvious that progress is made" is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.
Research Notes
Foundational metascience paper for evaluating psi research credibility. The impediments Ioannidis catalogues — low power, rare replication, publication bias, allegiance bias — are the same ones debated in parapsychology. Both critics (applying these to psi claims) and proponents (applying them to mainstream null results) invoke this analysis.
Self-correction is widely assumed to be a defining hallmark of science, but how often does it actually occur? Reviewing empirical evidence from psychology and biomedicine, Ioannidis argues that self-correction requires active replication effort — yet only ~1% of psychology papers are replications, fewer than 0.2% are independent direct replications, and most yield confirming results (perpetuated fallacies). With average power of 35% and modest bias, unchallenged fallacies may constitute 30–95% of published significant findings. A taxonomy of six discovery–replication paradigms quantifies the problem. Thirteen impediments to self-correction are catalogued, including publication bias, selective reporting, underpowered studies, and editorial bias against replication. Proposed reforms each carry unintended risks unless pursuit of truth remains the overriding priority.
Links
Related Papers
Cites
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data — Fanelli, Daniele (2009)
- The "File Drawer Problem" and Tolerance for Null Results — Rosenthal, Robert (1979)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling — John, Leslie K (2012)
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? — Pashler, Harold (2012)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan (2012)
Companion
- Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology — Meehl, Paul E (1978)
- Small Telescopes: Detectability and the Evaluation of Replication Results — Simonsohn, Uri (2015)
- Options for Prospective Meta-Analysis and Introduction of Registration-Based Prospective Meta-Analysis — Watt, Caroline A (2017)
- Scientists behaving badly — Martinson, Brian C (2005)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Ioannidis, John P.A (2012). Why Science Is Not Necessarily Self-Correcting. Perspectives on Psychological Science. https://doi.org/10.1177/1745691612464056
@article{ioannidis_2012_science_not_self_correcting,
title = {Why Science Is Not Necessarily Self-Correcting},
author = {Ioannidis, John P.A},
year = {2012},
journal = {Perspectives on Psychological Science},
doi = {10.1177/1745691612464056},
}