Information theory

Working Definition

Information theory is the mathematical theory of transmitting and storing messages, measuring information as the reduction of uncertainty: the amount of information in an event grows with its improbability, and the entropy of a source is the expected value of that quantity per symbol. From this single measure follow hard physical and mathematical bounds: how far data can be compressed without loss (source entropy), how much can be sent reliably through a noisy channel (channel capacity), and what price must be paid for lossy compression. In everyday speech, “information” means meaningful knowledge about the world; in the technical sense, it is a content-free quantity measured in bits, and keeping that distinction clear decides whether the concept explains anything or merely decorates prose.

Genesis and Global Lineage

Claude E. Shannon unveiled the theory in a two-part paper, A Mathematical Theory of Communication, in the Bell System Technical Journal (July and October 1948), solving a practical engineering problem at Bell Labs: how to measure how much a telegraph, telephone, or radio link can carry given a fixed bandwidth and noise floor. His immediate predecessors were Harry Nyquist (1924) and Ralph Hartley (1928), who already measured the “amount of information” logarithmically but without probability distributions; Shannon introduced a probabilistic source model, the entropy measure, and the fundamental coding theorems, drawing mathematically on statistical mechanics and Markov processes. In the same year, Norbert Wiener defined information in Cybernetics (1948) within the context of control and signal filtering. The 1949 book edition, featuring an introductory essay by Warren Weaver, stretched Shannon’s framework to questions of semantic meaning and communicative effectiveness—moves Shannon himself had carefully avoided—and from that moment the concept began a second life across linguistics, molecular biology, social science, and philosophy, often as a metaphor detached from any calculation. The technical lineage evolved differently: Andrey Kolmogorov (1965) shifted the focus from probability ensembles to the length of the shortest program that generates an individual object, while modern machine learning returned directly to Shannon via compression and cross-entropy loss.

Central European Lineage

The regional lineage here is formal and institutional rather than national. In §6 of Part I of his 1948 paper, Shannon remarks that the mathematical form of his function H = -\sum p_i \log p_i will be recognized by anyone familiar with statistical mechanics, pointing explicitly to “Boltzmann’s famous H theorem”—formulated by Ludwig Boltzmann, who studied and earned his doctorate at the University of Vienna (1866) and developed the H-theorem (1872) and the statistical definition of entropy (1877) between Graz and Vienna. Earlier, in 1929, Leó Szilárd—who began his engineering studies in Budapest before completing his doctorate in Berlin—analyzed Maxwell’s demon and demonstrated that acquiring one bit of information about a physical system carries an inescapable thermodynamic entropy cost, anticipating the link between information and entropy nineteen years before Shannon (though without documented influence on him). A well-known anecdote reported by Myron Tribus and Edward McIrvine (1971) holds that the very choice of the word “entropy” was suggested to Shannon at Princeton around 1940 by John von Neumann, educated at Budapest’s Lutheran Fasori Gymnasium and the University of Budapest, who joked that “nobody knows what entropy really is, so in a debate you will always have the advantage.” A later theoretical counterpoint came from Warsaw: Marian Mazur (1909–1983), an electrical engineer trained at the Warsaw University of Technology in interwar Poland and a leading figure of postwar Polish cybernetics, argued in Jakościowa teoria informacji (Qualitative Theory of Information, 1970) and Cybernetyka i charakter (1976) that Shannon had built a theory of signal quantity that says nothing about what information actually is, proposing instead a model of message transformations across a control loop.

Why Now

Two developments between 2018 and 2026 brought Shannon’s apparatus back to center stage: large language models and the physics of computation. Since 2018 frontier language models have been trained by minimizing cross-entropy loss, with performance reported in bits per token (or perplexity); empirical scaling laws (Jared Kaplan et al., 2020; Jordan Hoffmann et al., 2022) are essentially curves plotting how closely a neural network approaches the intrinsic entropy of natural language as compute budgets grow. The thesis that “compression is intelligence,” going back to Ray Solomonoff, has returned as a live research program, and the debate over whether a model that predicts the next token with low entropy “understands” the world is a replay of Weaver versus Shannon in modern dress. At the same time, the surging energy footprint of data centers has turned Landauer’s principle (1961) from a theoretical curiosity into an engineering horizon, while laboratory experiments with single-bit information engines in the 2010s and early 2020s physically confirmed Szilárd’s 1929 thought experiment. Meanwhile, public debates over “information overload” and disinformation lean on the everyday sense of the word, even as attention-economy recommendation engines optimize measurable information-theoretic quantities.

Explanatory Power

The concept separates what can be transmitted from what is worth transmitting—and proves that the former obeys hard bounds independent of hardware. First, it explains linguistic redundancy: written English is roughly 50 percent redundant, which is why we can read text riddled with typos and why lossless ZIP compression works; Shannon estimated the entropy of English experimentally by asking human subjects to guess successive letters in a passage. Second, it explains why the genetic code, cellular networks, and hard drives all rely on error-correcting codes: the noisy-channel coding theorem proves that arbitrarily reliable transmission is achievable at any rate below channel capacity, and impossible above it, at the price of coding delay. Third, it provides a general measure of statistical dependence that assumes no linearity—mutual information—used everywhere from computational neuroscience to side-channel leak detection. Finally, it shows why “more data” does not mean “more information”: if incoming data are highly predictable, they carry almost zero bits of surprise, explaining why billions of repetitive social-media posts do not shift a well-calibrated observer’s beliefs.

Axes of Debate

Underappreciated Angle

The least appreciated aspect of information theory is that its crowning achievements are impossibility theorems. Popular accounts remember the bit and entropy, forgetting that Shannon’s deepest results state what cannot be done: you cannot compress a source losslessly below its entropy, you cannot transmit reliably above channel capacity, and you cannot simultaneously achieve lower rate and lower distortion than the rate–distortion curve permits. That makes information theory one of the rare scientific frameworks that is socially useful primarily by saying “no”: every pitch for hundred-fold “lossless” video compression, zero-latency error-free wireless links, or an AI pipeline that extracts more signal from a dataset than the data contain can be dismissed without inspecting the code. Mazur was right that Shannon’s theory refuses to say what information means—yet it is precisely because of that refusal that it can state with mathematical certainty how much information is not there.