📌 Appears in:
Plain English Summary
Published in the APA's prestigious Psychological Methods, Kennedy lays down a challenge: if you want your psi study to actually prove something, you need to be able to show your idea is wrong, not just right. That's what "falsifiable" means -- designing experiments where failure is a real possibility. His framework combines several statistical tools and lands on a striking requirement: for the small effects typical in psi research, you'd need roughly 860 to 1,084 participants to run a properly powered study. That's a big ask! He also argues that simply claiming "any tiny effect counts" makes a hypothesis untestable, and that looking back at old studies to build your case is exploratory at best. Instead, researchers should plan ahead and spell out exactly what would count as success, failure, or an unclear result.
Actual Paper Abstract
Falsifiable research is a basic goal of science and is needed for science to be self-correcting. However, the methods for conducting falsifiable research are not widely known among psychological researchers. Describing the effect sizes that can be confidently investigated in confirmatory research is as important as describing the subject population. Power curves or operating characteristics provide this information and are needed for both frequentist and Bayesian analyses. These evaluations of inferential error rates indicate the performance (validity and reliability) of the planned statistical analysis. For meaningful, falsifiable research, the study plan should specify a minimum effect size that is the goal of the study. If any tiny effect, no matter how small, is considered meaningful evidence, the research is not falsifiable and often has negligible predictive value. Power ≥.95 for the minimum effect is optimal for confirmatory research and .90 is good. From a frequentist perspective, the statistical model for the alternative hypothesis in the power analysis can be used to obtain a p value that can reject the alternative hypothesis, analogous to rejecting the null hypothesis. However, confidence intervals generally provide more intuitive and more informative inferences than p values. The preregistration for falsifiable confirmatory research should include (a) criteria for evidence the alternative hypothesis is true, (b) criteria for evidence the alternative hypothesis is false, and (c) criteria for outcomes that will be inconclusive. Not all confirmatory studies are or need to be falsifiable.
Research Notes
Published in APA's Psychological Methods, this formalizes Kennedy's decade-long argument that psi research must adopt falsifiable designs with prespecified minimum effect sizes. Directly extends Kennedy (2016) and cites the Transparent Psi Project and Maier et al. (2020) Bem replication as exemplars. Sets a bar that would require N > 1,000 for most psi paradigms.
Falsifiable research requires that study designs can provide evidence a hypothesis is false as well as true. This article integrates power analysis, equivalence testing, Bayesian operating characteristics, and preregistration into a framework for falsifiable confirmatory research. Power >= .95 for a prespecified minimum effect size is optimal; .90 is good. If any nonzero effect is considered meaningful, the hypothesis is unfalsifiable. For d = 0.20, sample sizes of 858-1,084 are needed at adequate power. The alternative hypothesis can be rejected via noncentral t distributions when power is high. Preregistrations should specify criteria for evidence the hypothesis is true, false, or inconclusive. Retrospective meta-analyses are exploratory; prospective meta-analysis is preferred.
Links
Related Papers
Cites
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- A Preregistered Multi-Lab Replication of Maier et al. (2014, Exp. 4) Testing Retroactive Avoidance — Maier, Markus A (2020)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan (2012)
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
Extends
Same Research Program
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E (2024)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J.E (2014)
- Experimenter Fraud: What Are Appropriate Methodological Standards? — Kennedy, J.E (2017)
- Why Is Psi So Elusive? A Review and Proposed Model — Kennedy, James E (2001)
- The Capricious, Actively Evasive, Unsustainable Nature of Psi: A Summary and Hypotheses — Kennedy, J.E (2003)
- Conclusions about Paranormal Phenomena — Kennedy, J.E (2013)
- Is the Methodological Revolution in Psychology Over or Just Beginning? — Kennedy, J.E (2016)
- A Proposal and Challenge for Proponents and Skeptics of Psi — Kennedy, J.E (2004)
- Options for Prospective Meta-Analysis and Introduction of Registration-Based Prospective Meta-Analysis — Watt, Caroline A (2017)
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
Experimental evidence of non-classical brain functions
Self-Ascribed Paranormal Ability: Reflexive Thematic Analysis
📋 Cite this paper
Kennedy, James E (2024). Planning Falsifiable Confirmatory Research. Psychological Methods. https://doi.org/10.1037/met0000639
@article{kennedy_2024_falsifiable_research,
title = {Planning Falsifiable Confirmatory Research},
author = {Kennedy, James E},
year = {2024},
journal = {Psychological Methods},
doi = {10.1037/met0000639},
}