Red-teaming a language model is not the same as testing a web application, and treating it like one is how findings get missed. A web app has a defined set of inputs and a deterministic response. A model has an effectively infinite input space and a different answer each time you ask. This is a guide to approaching that problem methodically rather than by poking at the chat box until something interesting happens.
Start by Mapping, Not Attacking
Before a single adversarial prompt, understand the system. What is the model allowed to do? What tools can it call, what data can it reach, what actions can it take without a human? Where does it read text it did not generate: uploads, web browsing, retrieved documents, previous conversation? What is it explicitly meant not to do, and what would count as a breach? The attack surface of an AI system is defined by its permissions and its inputs, and you cannot test what you have not mapped.
The Categories Worth Testing
Instruction bypass. Can the model be talked out of its rules, directly or through role-play, hypotheticals, or encoding the request so it slips past the guardrails? The goal is to establish whether the safety behaviour is robust or merely a thin layer of surface training.
Indirect prompt injection. Plant instructions in the content the model ingests, a document, a page, a record, and see whether it acts on them when a normal user makes a normal request. This is where the serious findings usually are.
Data disclosure. Can you extract the system prompt, other users’ data, or fragments of training data? Can you get the model to reveal the contents of documents it has access to but the current user should not see?
Tool and action abuse. If the model can call tools, can you make it call them in ways the designer did not intend: querying records it should not, sending messages, chaining actions toward an outcome nobody authorised?
Excessive agency. Where the model acts autonomously, how far can a single manipulated instruction propagate before anything stops it? The question is not only can it be fooled, but how much damage a fooling can do.
Test the Consequence, Not the Sentence
It is easy to get a model to say something it should not and call it a finding. Whether that matters depends entirely on what the model can do. A chatbot that can be coaxed into rude language is a reputational nuisance. The same weakness in a model wired to a payments API is a financial control failure. Rate every finding by the consequence it enables in this specific deployment, not by how clever the prompt was. A jailbreak with no capability behind it is a curiosity; a mild bypass in front of real tools is critical.
Account for Non-Determinism
A model may refuse an attack nine times and comply the tenth. A single success is a real finding, not noise, because an attacker only needs it to work once. But a single failure proves nothing. Test each case several times, record how often it succeeds, and report the rate. Reproducibility here is statistical, not binary, and your report should say so plainly rather than pretending to a certainty the system does not have.
Writing It Up
A good AI red-team report ties each finding to a business consequence and a concrete mitigation, usually a matter of reducing permissions or inserting a human check rather than of retraining the model. Aim for defences the client can actually implement this quarter, because the model’s own behaviour is the part they can change least.
The Short Version
Red-team a model by mapping its permissions and inputs first, then working through instruction bypass, indirect injection, data disclosure, tool abuse and excessive agency. Rate findings by the consequence the deployment allows, not by the cleverness of the prompt. Test repeatedly and report success rates, because these systems are non-deterministic and one success in ten is still a way in.