Red Teaming (AI)

AI red teaming is structured adversarial probing of your model or agent, modeled on cybersecurity red teaming. Red teamers attempt to make the system produce harmful output, leak information, exceed its scope, or fail safety constraints. They use known attack patterns (jailbreaks from public datasets), invent new ones, and document each successful attack so the system can be hardened. Anthropic and OpenAI both run extensive red teams on every frontier model release. Production agent operators should run red-team exercises before launch and quarterly thereafter.

Run a red-team pass before any consumer-facing agent launch. Use both automated red-team tooling (Garak, Promptfoo) and human red-teamers familiar with the domain.

• NIST AI RMF: Adversarial testing
• Garak: LLM red-team tool
• Anthropic: Red-teaming research

Related Terms