AI Architecture #115
Scaling laws
original term: scaling laws (Eng.) · neural scaling laws, prawa skalowania sieci neuronowych, hipoteza skalowania
Global-circulation concept — no documented lineage in Central Europe. jump to documentation ↓
Working Definition
Neural scaling laws are empirical power-law relationships that describe how the performance of a machine-learning model—typically measured by its cross-entropy loss on held-out test data—improves predictably as a function of three core inputs: the number of model parameters (N), the size of the training dataset (D), and the total volume of floating-point compute (C) expended during training, provided the other two factors are not bottlenecking. Here “law” carries the meaning of an experimental regularity observed across many orders of magnitude (analogous to Kleiber’s law in biology or Moore’s law in semiconductors) rather than a theorem deduced from first principles: the exponents are fitted empirically to training runs rather than derived from a settled mathematical theory. In its strict technical sense, a scaling law states only that test loss follows a smooth, straight line on a log-log plot, allowing engineers to extrapolate the final loss of a massive model from cheap, small-scale pilot runs. In venture-capital and popular discourse, however, the concept is routinely conflated with the much stronger “scaling hypothesis”: the philosophical bet that simply pouring more compute and data into existing architectures is sufficient to unlock human-level general intelligence.
Greater compute and data can improve results along observed empirical trends, though extrapolating beyond measured regimes remains uncertain.
AI-assisted illustration · conceptual metaphor View in visual quiz →Genesis and Global Lineage
Power-law scaling has a long pedigree in statistical physics and biology, but entered deep learning as an empirical surprise contradicting classical learning theory—which warned that networks with far more parameters than data points should overfit. In December 2017, Baidu Research (Joel Hestness et al.) showed that generalization error across translation, language modeling, vision, and speech fell as a power law with dataset size. The canonical formulation arrived in January 2020 from OpenAI’s Jared Kaplan, Sam McCandlish, Dario Amodei, and colleagues (Scaling Laws for Neural Language Models): for autoregressive Transformers, test loss scales as a power law across more than seven orders of magnitude in compute, while depth-versus-width details shift loss only marginally. That paper underwrote GPT-3 (May 2020) and triggered a race toward trillion-parameter models. In March 2022, Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and colleagues at DeepMind published a crucial correction—the “Chinchilla scaling laws” (Training Compute-Optimal Large Language Models): because Kaplan et al. had used a fixed learning-rate schedule that undervalued extra data, they had scaled parameters much faster than tokens; DeepMind proved that for compute-optimal training, parameters and training tokens must grow in equal 1:1 proportion (roughly 20 tokens per parameter), showing that a 70B-parameter model (“Chinchilla”) trained on 1.4T tokens outperformed DeepMind’s 280B Gopher at the same compute cost.
Central European Lineage
We found no documented link between this concept and the intellectual history or historical experience of Central Europe.
We found no documented link between this concept and the intellectual history or historical experience of Central Europe (by our methodology, this means “none found,” not “none exists”; as of October 2026).
Why Now
Between 2020 and 2026, neural scaling laws became the coordinating metric of capital markets, semiconductor geopolitics, and statutory AI regulation. When OpenAI released GPT-4 in March 2023, its technical report noted that the team had accurately predicted final training loss in advance from runs using 1{,}000\times to 10{,}000\times less compute. Policymakers adopted training compute as their regulatory proxy: the October 2023 U.S. Executive Order set reporting mandates at 1026 FLOPs, while Article 51 of the EU AI Act (2024) set 1025 FLOPs as the statutory threshold for “systemic risk.” Simultaneously, between 2024 and 2026, pretraining scaling collided with physical and informational limits: Epoch AI projected that public human web text would be exhausted between 2026 and 2032 (the “data wall”), while multi-gigawatt data centers strained electrical grids. In response, late 2024 and 2025 saw the industry pivot to a second curve—inference-time (test-time) scaling laws, exemplified by OpenAI’s o1/o3 and DeepSeek-R1—where models improve on reasoning benchmarks by expending more compute generating and verifying internal chains of thought at query time rather than solely inflating pretraining size.
Explanatory Power
Scaling laws transform recent AI history from a tale of mysterious algorithmic flashes into a problem of industrial resource allocation. First, they explain the financial audacity of AI labs: no board would commit $500 million to a three-month GPU run if deep learning were a lottery; scaling laws act like wind-tunnel physics, letting engineers run miniature experiments, fit the power-law exponent, and predict the giant run’s loss before lighting up the cluster. Second, they explain the post-2022 pivot from bragging about parameter counts to hoarding data and signing publisher licensing deals: Chinchilla proved the industry had built oversized, underfed networks, triggering a scramble for trillions of tokens and pushing open-weight teams (like Meta’s Llama 3 in 2024, trained on 15T tokens even at 8B parameters) to “over-train” smaller models well past Chinchilla optimality to slash inference costs. Third, they explain why export controls on advanced GPUs and EUV lithography became the centerpiece of US–China tech policy: if capability is a predictable function of compute FLOPs, restricting silicon throughput directly constrains model capability.
Axes of Debate
Smooth power-law predictability versus abrupt “emergent” jumps
Researchers led by Jason Wei et al. at Google (Emergent Abilities of Large Language Models, 2022) documented dozens of tasks—such as multi-digit arithmetic and multi-step reasoning—where accuracy stays near random chance across small models and shoots up abruptly past a critical scale threshold, arguing that smooth cross-entropy loss conceals phase-transition-like capability leaps. Strongest counterargument: In a NeurIPS 2023 Best Paper (Are Emergent Abilities of Large Language Models a Mirage?), Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo showed that over 92% of reported “emergent” jumps vanish when discontinuous, all-or-nothing evaluation metrics (like exact-string match) are replaced with continuous metrics (like per-token log-likelihood), proving token probabilities improve smoothly all along. Nature of the dispute: substantive — continuous token-level loss is smoothly predictable, yet real-world users often care precisely about discontinuous, all-or-nothing task success (code either compiles and passes tests, or fails).
“Scale is all you need” versus the data wall and diminishing returns
Champions of Richard Sutton’s Bitter Lesson (2019) and frontier lab leaders argue that scaling compute—first in pretraining, and since 2024 in test-time search—overcomes every apparent ceiling. Skeptics (from Gary Marcus to systems researchers tracking the exhaustion of human web text, Villalobos et al., 2022) argue that because power laws require an exponential 10\times increase in compute and energy for each linear step down in loss, pretraining scaling faces diminishing economic returns, copyright exhaustion, and synthetic-data inbreeding (model collapse). Strongest counterargument: Even if next-token cross-entropy loss continues to inch down along a power law, scaling laws say nothing about how a 0.05-nat reduction in perplexity translates into economic value or factual reliability; a model can improve on the log-loss curve while remaining too unreliable for autonomous deployment in medicine or law. Nature of the dispute: mixed — the exponential cost of power-law exponents and finite human text are empirical facts, while whether synthetic data and test-time compute can carry the curve to AGI is a multi-hundred-billion-dollar bet.
Fundamental law of nature versus contingent artifact of web data
Physicists turned ML researchers (Jared Kaplan, Yasaman Bahri, Surya Ganguli) explain scaling exponents as universal properties of high-dimensional data manifolds. Data-curation researchers (Surya Gunasekar et al. with Microsoft’s Phi series, 2023; Sorscher et al., Beyond Neural Scaling Laws, 2022) counter that exponents are not constants of nature: aggressive data pruning and textbook-quality curation can bend the curve, letting a 3B model match a 50B model trained on raw web scrapes. Strongest counterargument: While better curation shifts the constant prefactor and improves sample efficiency, once a curated distribution is fixed, the relationship between compute and loss still reverts to a power law—showing data quality and scale act as multipliers rather than substitutes. Nature of the dispute: substantive — an active scientific investigation into why optimization over natural data exhibits scale-free power-law geometry.
Underappreciated Angle
The least appreciated function of neural scaling laws is sociological and institutional: they operate as a coordination technology for massive capital and regulatory bureaucracy, just as Moore’s Law did for semiconductors. Sociologists of technology observed that Moore’s Law was not a passive law of silicon physics, but a self-fulfilling industrial rhythm: because chipmakers, equipment suppliers (like ASML), and lenders believed the two-year doubling curve would hold, they synchronized ten-year R&D roadmaps and credit lines to make it come true. Neural scaling laws play the exact same coordinating role today. Because a log-log plot of test loss versus FLOPs looks as deterministic as thermodynamics, it gives corporate boards, sovereign wealth funds, power utilities, and Brussels lawmakers a shared quantitative ruler (1025 FLOPs) on which to sign nuclear-power agreements, underwrite $100 billion data centers, and draft statutory thresholds before the models exist—starving alternative, non-scaling research paradigms of capital until physical limits of energy and data force the industry to redefine “scaling” to keep the line pointing down.
One-Sentence Takeaway
Neural scaling laws do not prove that bigger models become wiser; they prove only that next-token prediction loss falls along a predictable power-law curve—and everything else, from “emergent reasoning” to AGI, is an interpretation that investors and researchers have projected onto that line.
Related Concepts Network
How this concept reinforces, constrains, or complements other analytical frameworks in the corpus. View on graph →
-
Large language models (LLMs) reinforces
Scaling laws provide the economic and epistemic justification for the existence of large language models: they explain why spending hundreds of millions of dollars to train ever-larger networks is a predictable bet rather than a blind gamble.
-
Transformer complements
Smooth power-law loss curves were measured primarily on the Transformer architecture; how much belongs to deep learning in general versus the specific properties of self-attention remains an active theoretical question.
-
Emergence in tension
Abrupt jumps in downstream task accuracy at specific scale thresholds challenge the picture of smooth, predictable power-law improvement—sparking an intense debate over whether emergence is a physical phase transition or a metric artifact.
-
Goodhart’s law constrains
Cross-entropy test loss decreases predictably along the scaling curve, but test loss is not identical to real-world usefulness or safety; optimizing purely for the curve can diverge from human goals.
-
Model collapse constrains
If the supply of high-quality human-generated data is finite and training on synthetic outputs degrades distribution tails, the data axis of scaling laws faces a hard ecological ceiling.
-
Frontier AI reinforces
Regulatory frameworks for frontier AI rely on floating-point compute thresholds (such as 10^25 or 10^26 FLOPs) precisely because scaling laws establish training compute as a proxy predictor of downstream capability.
-
Artificial general intelligence (AGI) complements
The “scaling hypothesis” is the philosophical bet that extrapolating empirical scaling curves far enough will arrive at general intelligence; scaling laws themselves measure only test loss and do not prove that bet.
Reading Canon & Translation Bridge
Foundational texts, introductions, and critical perspectives—with availability of English and Polish editions.
-
foundational text 2020 · English
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, Scaling Laws for Neural Language Models
The January 2020 OpenAI paper that gave the concept its canonical modern form: cross-entropy loss in Transformer language models falls as a smooth power law with parameter count, dataset size, and training compute across more than seven orders of magnitude, while architectural depth-versus-width details matter only marginally.
In English: original text written in English
In Polish: no Polish edition
-
foundational text 2022 · English
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch i in., Training Compute-Optimal Large Language Models
DeepMind's landmark correction to Kaplan et al., universally known as the “Chinchilla paper”: showed that for a fixed compute budget, model parameters and training tokens should be scaled in equal proportions—proving that 2020–2021 giants like GPT-3 and Gopher were severely undertrained relative to their size.
In English: original text written in English
In Polish: no Polish edition
-
introduction & synthesis 2017 · English
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, Yanqi Zhou, Deep Learning Scaling is Predictable, Empirically
An earlier December 2017 study from Baidu Research demonstrating power-law generalization error curves across machine translation, language modeling, image classification, and speech recognition, while noting that their exponents lacked a theoretical derivation.
In English: original text written in English
In Polish: no Polish edition
-
critical perspective 2023 · English
Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, Are Emergent Abilities of Large Language Models a Mirage?
A NeurIPS 2023 Best Paper arguing that seemingly abrupt “emergent” capability jumps disappear when discontinuous evaluation metrics (like exact-match accuracy) are replaced with continuous metrics, restoring smooth power-law predictability.
In English: original text written in English
In Polish: no Polish edition
-
critical perspective 2022 · English
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn, Will we run out of data? Limits of LLM scaling based on human-generated data
The influential Epoch AI study estimating that the global stock of high-quality public human text will be fully exhausted by frontier training runs between 2026 and 2032, establishing a physical ceiling on the data axis of Chinchilla scaling.
In English: original text written in English
In Polish: no Polish edition
-
cross-disciplinary 2019 · English
Richard S. Sutton, The Bitter Lesson
A short, influential essay by a pioneer of reinforcement learning: seventy years of AI history teach that general methods leveraging massive computation invariably defeat handcrafted human domain knowledge—the philosophical manifesto behind the scaling hypothesis.
In English: original text written in English
In Polish: Polish edition not verified
-
cross-disciplinary 2017 · English
Geoffrey West, Scale: The Universal Laws of Growth, Innovation, Sustainability, and the Pace of Life in Organisms, Cities, Economies, and Companies
A Santa Fe Institute physicist's synthesis of allometric power laws in biology (such as Kleiber's law linking metabolic rate to body mass) and urban systems, placing neural scaling laws within a much older scientific tradition of empirical power-law regularities.
In English: original text written in English
In Polish: Polish edition not verified
Primary-Source Quotes
The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.
for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.