Causal inference

Working Definition

Causal inference is the family of mathematical formalisms and research designs that allow scientists to distinguish passive statistical association from the actual effect of an active intervention. Instead of asking how the probability distribution of outcome Y looks when we merely observe X, it asks how Y would shift if we stepped in from outside and forced X to take a specific value—or what would have happened in an individual case had a past event gone differently (counterfactuals). In everyday conversation the topic is often reduced to the paralyzing taboo “correlation does not imply causation”; in its technical sense, causal inference specifies the exact mathematical conditions under which a causal effect can be identified by combining data with explicit, falsifiable structural assumptions.

Genesis and Global Lineage

For most of the twentieth century, mainstream academic statistics—shaped by Karl Pearson, who in the 1911 edition of The Grammar of Science denounced “cause and effect” as a pre-scientific fetish and sought to replace it with contingency tables—shied away from causal vocabulary outside randomized experiments pioneered by Jerzy Neyman (1923) and Ronald A. Fisher (1925). Breaking that taboo required three independent revolutions. First, in econometrics, Trygve Haavelmo's simultaneous-equation models in the 1940s and the later “credibility revolution” of Joshua Angrist, Guido Imbens, and David Card developed quasi-experimental designs (instrumental variables, difference-in-differences, regression discontinuity) to extract causal signals from natural experiments. Second, in biostatistics and sociology, Donald B. Rubin (1974) generalized Neyman's notation into the “potential outcomes” framework (the Neyman–Rubin Causal Model), framing causation as a missing-data problem over unobservable counterfactuals. Third, in artificial intelligence, Judea Pearl in the 1990s married Bayesian networks with structural equations to create Directed Acyclic Graphs (DAGs) and the do(x) calculus. Across twentieth-century social science, these distinctions were routinely flattened into multiple regression, where researchers mistakenly believed that tossing more “control variables” into an equation automatically isolated a causal effect.

Central European Lineage

The formal mathematical foundation of the potential outcomes model was created in Polish between Bydgoszcz and Warsaw. Jerzy Spława-Neyman, trained in mathematics at Kharkov University (now Kharkiv, Ukraine) from 1912 to 1917 under probability theorist Sergei N. Bernstein, moved to newly independent Poland in 1921 to work as a statistician at the State Agricultural Research Institute (Państwowy Instytut Naukowo-Rolniczy) in Bydgoszcz. In his 1923 Polish treatise Próba uzasadnienia zastosowań rachunku prawdopodobieństwa do doświadczeń polowych (On the Application of Probability Theory to Agricultural Experiments), published in Roczniki Nauk Rolniczych and submitted toward his 1924 doctorate at the University of Warsaw, Neyman introduced a doubly indexed array of unknown “potential yields” U_ij—the yield that crop variety i would produce if planted on field plot j. As Guido Imbens and Donald Rubin note, this notation was the first formal mathematical statement of potential outcomes. Neyman saw clearly that on any plot j in a single harvest, only one variety can be planted and observed, leaving alternative yields counterfactual, and derived the variance of the difference-in-means estimator under sampling without replacement. When Dorota M. Dąbrowska and Terence P. Speed translated Section 9 in Statistical Science (1990) alongside Donald B. Rubin's commentary, Rubin showed that while Neyman had transmitted the notation into English experimental theory by 1935, the 1923 Bydgoszcz paper contained the formal germ of modern counterfactual statistics half a century before Rubin extended it to observational studies in 1974.

Why Now

Between 2018 and 2026, causal inference moved from a specialist methodological debate to the center of science and AI through three developments. First, Judea Pearl and Dana Mackenzie published The Book of Why (2018), popularizing the three-rung “Ladder of Causation”—Rung 1: Association (P(Y | X), passive seeing), Rung 2: Intervention (P(Y | do(X)), active doing), and Rung 3: Counterfactuals (P(Y_x | x', y'), retrospective imagining)—just as “Big Data” hype claimed that raw correlations made scientific theory obsolete. Second, in October 2021, the Nobel Memorial Prize in Economic Sciences was awarded to David Card, Joshua Angrist, and Guido Imbens for analyzing causal relationships through natural experiments. Third, the rise of Large Language Models since 2022 exposed the practical boundary between Rung 1 and Rung 2: foundation models trained on next-token prediction capture high-dimensional correlational patterns, yet hallucinate causal direction, fail under distribution shifts when policies intervene in the system, and cannot separate a symptom from a cause without explicit causal scaffolding.

Explanatory Power

Causal inference explains why higher predictive accuracy can produce catastrophic policy decisions, and why blindly “controlling for everything” in a regression can manufacture a false effect (collider bias). First, in clinical AI, it resolves the hospital pneumonia paradox: models trained on health records learned that asthma lowered a pneumonia patient's risk of death and recommended sending asthmatics home. Historically, asthmatic pneumonia patients were triaged straight into intensive care, and aggressive ICU treatment lowered their mortality; using that correlation as a treatment rule confuses the effect of an intervention with baseline disease risk. Second, in labor economics, Card and Krueger's difference-in-differences comparison of fast-food restaurants across the New Jersey–Pennsylvania border isolated the causal effect of a minimum-wage hike from macroeconomic trends. Third, in digital advertising, causal models separate organic purchase intent from ad lift: users shown retargeted ads buy at high rates mainly because auctions target people already about to check out, so naive correlation wildly overstates ad effectiveness.

Axes of Debate

Underappreciated Angle

Popular accounts celebrate causal inference as a mathematical machine for extracting certainty from messy data, yet its deepest theoretical achievement is a set of impossibility theorems—a proof of what data alone can never tell us. Judea Pearl's Ladder of Causation has the logical structure of a limitation theorem akin to Gödel's in logic: no amount of passive observational data from Rung 1, even with an infinite sample and a trillion-parameter neural network, can ever answer a Rung 2 intervention question or a Rung 3 counterfactual question unless the investigator supplies qualitative causal assumptions from outside the dataset. Equally counterintuitive is collider bias: generations of researchers were taught that “controlling for more variables” always makes a regression safer, whereas causal graph theory proves that conditioning on a common effect of two variables actively injects a phantom correlation. In an era tempted to replace scientific theory with brute-force statistical scaling, causal inference offers the hardest mathematical proof that understanding the world requires models of mechanisms, not just denser tables of co-occurrence.