In-context learning

Working Definition

In-context learning (ICL) is the ability of a pretrained sequence model—most notably a Transformer language model—to adapt to and perform a novel task at inference time solely by conditioning on instructions or input–output demonstrations placed inside its active context window, without any gradient updates or changes to its weights. The model is not fine-tuned; whatever “learning” occurs happens entirely inside the temporary activation states of a single forward pass and vanishes the moment the context window is cleared. In its precise technical sense, the term designates an emergent computational mechanism whose internal implementation (Bayesian concept selection, induction-head circuits, or implicit gradient descent) is a central object of theoretical machine-learning research. In everyday practitioner talk, it is frequently conflated with few-shot prompting—that is, the user’s act of pasting examples into a prompt, rather than the surprising fact that a frozen network can execute an inner learning loop without updating parameters.

Genesis and Global Lineage

The term was coined by Tom B. Brown and his OpenAI co-authors in the May 2020 GPT-3 paper Language Models are Few-Shot Learners, deliberately replacing earlier, ambiguous phrases like “zero-shot transfer” that confused the absence of gradient updates with the absence of task examples. In Footnote 1 and Figure 1.1 of that paper, the authors framed unsupervised pretraining across the internet as the “outer loop” of meta-learning, and defined in-context learning as the “inner loop” occurring within the activations of each forward sequence. The broader idea that a neural network with fixed weights could learn new tasks dynamically from its inputs had deep roots in earlier meta-learning research—most notably Jürgen Schmidhuber’s “Fast Weight Programmers” (1992) and Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell’s work on meta-learning with recurrent LSTMs (2001)—and had been prefigured in OpenAI’s GPT-2 paper (Radford et al., 2019), which showed that natural-language prompts could specify tasks zero-shot. What made May 2020 a watershed was the empirical discovery that the gap between zero-shot and few-shot in-context performance widened dramatically as model scale increased from 125 million to 175 billion parameters. From 2021 onward, in-context learning evolved from an empirical observation into a major theoretical research program bridging statistical learning theory, mechanistic interpretability, and optimization.

Central European Lineage

We found no documented link between this concept and the intellectual history or historical experience of Central Europe.

Why Now

Between 2020 and 2026, in-context learning became the invisible foundation of how almost every consumer and enterprise interacts with artificial intelligence. Before 2020, deploying a language model like BERT for a new corporate classification or extraction task required collecting thousands of labeled examples, running GPU fine-tuning via backpropagation, and hosting a separate copy of the modified weights for every single task; thanks to in-context learning, a single frozen foundation model can serve ten thousand completely different downstream workflows simultaneously simply by swapping the text in the prompt prefix. Furthermore, between 2023 and 2026, the expansion of context windows from 2,048 tokens (in GPT-3) to 128,000, 1 million, and 2 million tokens (in Gemini 1.5/2.0 and Claude 3) transformed the scale of the phenomenon from “few-shot learning” (3 to 10 examples) to many-shot in-context learning, where users feed hundreds of examples, an entire codebase, or a whole textbook grammar of an endangered language into a single prompt. Simultaneously, landmark mechanistic studies between 2022 and 2024—identifying “induction heads” inside Transformer circuits (Olsson et al., 2022) and proving that linear attention layers can literally implement steps of gradient descent in their forward activations (von Oswald et al., 2023; Akyürek et al., 2023)—made in-context learning the premier Exhibit A in the scientific debate over whether large models merely memorize training data or synthesize genuine learning algorithms inside their weights.

Explanatory Power

The concept separates two capacities that popular commentary on AI persistently confuses: what a model stores in its permanent parameters versus what it can compute dynamically from its active working memory. First, it explains how a completely frozen network can adapt instantly to user-specific rules, private schemas, or newly invented notation that did not exist when pretraining ended—without suffering the “catastrophic forgetting” that plagues traditional gradient fine-tuning. Second, it illuminates the surprising findings of Sewon Min and colleagues (Rethinking the Role of Demonstrations, 2022), who showed that on many standard NLP benchmarks, randomly scrambling the ground-truth output labels in few-shot examples barely reduced accuracy: in those regimes, in-context demonstrations work primarily by telling the model the expected input distribution, the label space, and the output format, activating a task the model already knows from pretraining. Third, it unifies seemingly disparate engineering patterns—few-shot prompting, Retrieval-Augmented Generation (RAG), system prompts, and agentic scratchpads—as variations of a single phenomenon in which context-window activations function as temporary, per-request “fast weights.”

Axes of Debate

Underappreciated Angle

In-context learning is almost universally celebrated as a capability breakthrough, yet it is simultaneously the architectural root of the hardest security and evaluation problems in modern AI. Because ICL allows tokens inside the context window to act as temporary “fast weights” that reprogram the model’s forward pass on the fly, every piece of text entering the context window can alter what program the model executes. When a developer puts examples in a prompt, we call it in-context learning; when an external webpage, email, or PDF read by an AI agent contains hidden instructions or adversarial examples that hijack behavior, we call it indirect prompt injection or many-shot jailbreaking (shown by Anthropic in 2024, where pasting hundreds of fictional dialogues of a compliant assistant into a long context window systematically overrode RLHF safety training). In-context learning and prompt injection are the exact same forward-pass reprogramming mechanism used by different actors. Consequently, if a model’s effective runtime weights are co-determined by its current context, there is no such thing as auditing “the model” in isolation—every safety guarantee and capability score is conditional on the context distribution encountered in the wild.