Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010)
🧐 Skeptical/Critical ↗📌 Appears in:
Plain English Summary
Meta-analysis is a technique for pooling results from many studies to find the big picture -- and Ray Hyman thinks it can be a magician's trick when applied to psychic research. Responding to a study claiming strong evidence for telepathy in ganzfeld experiments (where a receiver in a relaxed, sensory-reduced state tries to pick up mental images from a sender), Hyman dug into the numbers and found something revealing. Nearly all the impressive hit rates came from just four researchers, while everyone else scored at chance. When the experiments used video clips as targets, results looked great, but with still images -- the kind used in the original studies -- performance was flat. Most damning: the newest, most rigorous round of experiments hit right at the 25% rate you'd expect from pure guessing. Hyman's takeaway: don't let averaging across messy, inconsistent data create an illusion of proof.
Abstract
Storm, Tressoldi, and Di Risio (2010) rely on meta-analyses to justify their claim that the evidence for psi is consistent and reliable. They manufacture apparent homogeneity and consistency by eliminating many outliers and combining databases whose combined effect sizes are not significantly different— even though these combined effect sizes consist of arbitrary and meaningless composites. At best, their study provides a recipe for conducting a replicable extrasensory perception experiment. This recipe includes following a design that employs the standard ganzfeld psi methodology and uses "selected" subjects. An experiment, having adequate power and that meets these criteria, has already been conducted and failed to produce evidence for psi. Parapsychology will achieve scientific acceptability only when it provides a positive theory with evidence based on independently replicable evidence. This is something it has yet to achieve after more than a century of trying.
Links
Related Papers
Same Research Program
- Anomaly or Artifact? Comments on Bem and Honorton — Hyman, Ray (1994)
- A Joint Communiqué: The Psi Ganzfeld Controversy — Hyman, Ray (1986)
- Evaluation of a Program on Anomalous Mental Phenomena — Hyman, Ray (1996)
- Parapsychological Research: A Tutorial Review and Critical Appraisal — Hyman, Ray (1986)
Cites
- Does Psi Exist? Replicable Evidence for an Anomalous Process of Information Transfer — Bem, Daryl J (1994)
- Why Is Psi So Elusive? A Review and Proposed Model — Kennedy, James E (2001)
- The Capricious, Actively Evasive, Unsustainable Nature of Psi: A Summary and Hypotheses — Kennedy, J.E (2003)
- An Assessment of the Evidence for Psychic Functioning — Utts, Jessica (1996)
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Hyman, Ray (2010). Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010). Psychological Bulletin. https://doi.org/10.1037/a0019676
@article{hyman_2010_meta_analysis_conceals,
title = {Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010)},
author = {Hyman, Ray},
year = {2010},
journal = {Psychological Bulletin},
doi = {10.1037/a0019676},
}