Red teaming

Working Definition

Red teaming is the structured practice of subjecting one’s own system, strategic plan, or machine learning model to attacks by an independent team tasked with thinking like a determined adversary in order to uncover failure modes that the designers failed to anticipate. In its technical and regulatory AI definition—codified by U.S. Executive Order 14110 (2023), NIST, and frontier laboratories such as Anthropic and Google DeepMind—it denotes a systematic pre-deployment testing effort to elicit harmful behaviors, security vulnerabilities, or dangerous capabilities (such as cyber-offense or biosecurity uplift) in a controlled environment. In everyday internet parlance, the phrase is used loosely for any ad-hoc attempt to “jailbreak” a chatbot into swearing or bypassing its content filters. Keeping the two apart is critical: rigorous red teaming operates with an explicit threat model, documentation protocol, and remediation loop, whereas casual jailbreaking aims only at a viral screenshot.

Genesis and Global Lineage

The practice originated in military wargaming, where one side role-plays the enemy: the color convention pitting a hostile “Red Team” against a defending “Blue Team” dates to nineteenth-century Prussian Kriegsspiel and was institutionalized in Cold War simulations at RAND and the U.S. Department of Defense. A second lineage comes from intelligence alternative analysis: after failing to anticipate the 1973 Yom Kippur War, Israel’s Military Intelligence (Aman) created a Devil’s Advocate unit (Ipcha Mistabra) mandated to argue contrarian hypotheses, and after 9/11 and the 2003 Iraq WMD assessments, the U.S. Army established the Red Team Leader Course at Fort Leavenworth. A third lineage arose in offensive cybersecurity (penetration testing, as in NATO CCDCOE’s annual Locked Shields exercises in Tallinn). Micah Zenko’s monograph Red Team (2015) synthesized these into simulations, vulnerability probes, and alternative analysis. While Microsoft formed an AI Red Team in 2018, the concept entered the generative AI canon in 2022 through two simultaneous papers: Ethan Perez et al. at DeepMind (Red Teaming Language Models with Language Models) trained one LLM to attack another, and Deep Ganguli et al. at Anthropic (Red Teaming Language Models to Reduce Harms) published 38,961 human adversarial conversations across model scales. Since 2023, AI labs and lawmakers have adopted it as the standard pre-release ritual for foundation models—often narrowing institutional self-critique into a hunt for prompt-level filter bypasses.

Central European Lineage

We found no documented link between this concept and the intellectual history or historical experience of Central Europe.

Why Now

Between 2022 and 2026, AI red teaming moved from internal lab experiment to mandatory international governance. In August 2023, at DEF CON 31 in Las Vegas, the White House co-sponsored the first public Generative AI Red Team Challenge, where thousands of participants exchanged over 164,000 adversarial prompts with models from eight AI companies. On October 30, 2023, President Biden’s Executive Order 14110 wrote a statutory definition of “AI red-teaming” into federal policy and ordered developers of foundation models above 10²⁶ FLOP to share red-team results with the government (and after the order’s revocation in January 2025, its technical definition remained in NIST’s July 2024 Generative AI guidance). In the EU, Article 55(1)(b) of the AI Act (2024, applicable August 2025) legally requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. Meanwhile, as labs published elaborate red-teaming taxonomies (such as Anthropic’s June 2024 framework covering CBRN experts, automated attacks, and multimodal probing), academic scrutiny grew: at AIES 2024, the Best Paper Award went to Michael Feffer, Zachary Lipton, Hoda Heidari, and coauthors at Carnegie Mellon for Red-Teaming for Generative AI: Silver Bullet or Security Theater?, showing the field still lacked consensus on what constitutes a rigorous audit.

Explanatory Power

The concept highlights the structural gap between “scoring 92% on a static safety benchmark” and “surviving forty hours of probing by someone paid to break it.” Standard benchmarks evaluate a fixed distribution chosen by creators; a red team searches for out-of-distribution corners where training assumptions fail. First, in DeepMind’s 2022 automated red-teaming study, Perez et al. found that their 280B-parameter chatbot not only generated offensive stereotypes when provoked, but spontaneously handed out real phone numbers of hospitals and private citizens memorized from its training corpus—a failure no multiple-choice benchmark tested for. Second, in Anthropic’s 2022 scaling study, Ganguli et al. discovered that models aligned via Reinforcement Learning from Human Feedback (RLHF) became progressively harder for human red teamers to break as parameter count grew, whereas models aligned via prompt engineering showed flat resistance curves across scale. Third, organizational red teaming (such as Gary Klein’s project premortem, where a team imagines its deployment has already caused a catastrophe and writes the history of how it happened) surfaces institutional blind spots that polite risk meetings suppress.

Axes of Debate

Underappreciated Angle

When the AI industry borrowed red teaming from the military and intelligence world, it quietly amputated the most important half of the doctrine. In Israeli intelligence after 1973 and U.S. Army doctrine at Fort Leavenworth, a red team was not a group of technicians hired to poke at hardware on a workbench; it was an institutional counterweight empowered to challenge the commander’s strategic framing—asking whether the mission’s assumptions were delusional and whether the operation should be aborted. As Subhabrata Majumdar, Brian Pendleton, and Abhishek Gupta note in Red Teaming AI Red Teaming (2025), AI red teaming has shrunk the practice from the organizational level down to the artifact level: external testers type clever prompts into a chat box for three weeks before a scheduled launch, without the mandate or governance authority to question the business model or cancel the release date. Moreover, because every exercise is bounded by the backgrounds of those recruited onto the team, the catalog of discovered flaws is never an objective map of total risk—it is a mirror of who was invited into the room. A red team permitted to break the refusal filter, but never permitted to halt the launch, performs only half the job for which the name was invented.