An AI red teaming methodology is a chain of custody, not a list of clever prompts. The prompts are the easy part, and they are the part every vendor demo shows you. What turns a week of poking at a chatbot into evidence a board can fund and an engineer can fix is the structure around it: a permission and data-flow map built before the first payload is sent, attack classes worked in an order where each one supplies the primitive the next needs, findings judged by what the system did rather than what the model said, and a write-up that survives a retest after the model version changes.
Most published guidance stops at the adversarial prompt catalogue. That catalogue is useful. It is not a methodology.
What a Red Team Methodology Has to Deliver for an AI System
There is a difference between attacking a model and assessing a system, and confusing them is the first mistake. The model is one component. Around it sit a system prompt, a retrieval layer, a set of tool definitions, the credentials those tools hold, an output channel and whatever downstream service acts on that output. The model is rarely the weakest element. The permissions wrapped around it usually are.
Jailbreak-only testing produces reports nobody can act on. A transcript where the model agrees to write malware, insult a customer or describe how to bypass a control is a behaviour observation. It tells you something about the surface training applied by the provider. It tells you almost nothing about your risk, because a forbidden sentence is not a security finding unless something followed from it. Reports that conflate the two lose engineering trust quickly, and once lost it is hard to get back.
Three outputs make an engagement defensible:
- A mapped attack surface: every channel the model reads from, every action it can take, every credential in play.
- Consequence-proven findings: a record changed, a message left the queue, data crossed a tenant boundary.
- Reproducible evidence: attempt counts, model identifiers, timestamps, tool call logs, canary artefacts.
This work sits alongside conventional application testing rather than replacing it. The authentication on the chat endpoint, the IDOR in the conversation history API, the stored XSS in rendered model output: all still yours, all still found by ordinary methods. The AI-specific layer is additive. Scope both, and be explicit in the scoping document about which parts of the stack are in play.
Map Permissions, Inputs and Data Flows Before You Send a Prompt
Nothing useful happens until the inputs are enumerated. The model reads from more places than the chat box: uploaded documents, retrieved chunks from a vector store, API responses, the output of one tool fed into the next turn, email bodies, ticket descriptions, calendar invites, web pages an agent fetches on request. Each of those is an untrusted input channel, whether or not the architecture diagram labels it as one.
Then write down the model’s effective permissions. Not the permissions of the person typing. The model’s effective permissions are the union of every credential each of its tools holds, and that union is almost always wider than anyone expects. A retrieval service authenticated as a service account with read access across a whole document store does not care that the asking user sits in marketing. An MCP connector configured once, by an engineer, with a token minted for convenience, inherits whatever that token can do. Teams routinely scope an engagement around the chat interface and never ask what the connector is authenticated as. That omission alone can invalidate the whole assessment.
Trace the flows outward as well. What can the system write, send, delete, pay, publish or escalate? Who sees the output, and is that output rendered as HTML, piped into a shell, or committed to a repository? An agent that can only answer questions has a disclosure problem. An agent that can send mail has an exfiltration problem. An agent that can approve something has an authorisation problem.
The deliverable from this phase is one page: a trust boundary sketch showing inputs, the model, its tools, their credentials and the downstream effects. It drives test selection and it stops the engagement drifting into prompt trivia, which is the natural failure mode when a tester with a week booked has nothing but a text box.
The Five Attack Classes, Worked in Dependency Order
Order matters because each class supplies the primitive the next one needs to prove impact. Work them out of sequence and you end up with interesting transcripts and no demonstrated consequence.
1. Instruction bypass
Establish what the system prompt actually forbids, then establish whether that boundary is enforced anywhere outside the model. Often it is not. The topic restriction, the refusal to discuss pricing, the instruction never to reveal internal policy: all of it is a request made in the same channel the user’s text arrives through. Finding the bypass is step one, not the finding itself. What you now have is a steering primitive.
2. Indirect injection
This is the class most assessments skip, and it is the class that matters most. Skipping it is not laziness, it is a scoping artefact: indirect injection needs setup. You have to get instructions into content the model ingests, which means a shared document, a ticket body, a calendar invite, a knowledge base article or a page the agent will fetch. A test plan that only permits typing into the chat window cannot reach it. Ask for write access to one of those sources during scoping, with a named test container, or accept that the highest-impact class goes untested. The underlying reason it works at all is covered in more depth in this analysis of why prompt injection defines LLM security.
3. Data disclosure
System prompt extraction, cross-tenant retrieval leakage, context contamination between sessions. The RAG case is the one that keeps appearing: a shared document store where chunk-level permissions were never applied, retrieval running as a service account, and a query phrased around another department’s project name returning content the asking user cannot open in the source system at all. The model did nothing wrong. The retrieval layer did.
4. Tool abuse and excessive agency
Chain the injection into an action. This is where the serious findings usually are. Watch for the confused deputy pattern: the agent holds authority the attacker does not, the attacker supplies the instruction, and the agent exercises its own authority on the attacker’s behalf. Classic worked example, and one worth writing up carefully: an internal support agent with a Jira connector, a crafted ticket body instructing it to summarise recent account notes and attach them to a reply addressed to an external contact. The agent is doing its job. It is just doing it for the wrong person. Agent tool use and its containment deserves its own test block whenever tools are in scope.
5. Consequence verification
Confirm the action landed. Not that the model claimed it landed. Models narrate success they did not achieve, cheerfully and in detail.
OWASP’s Top 10 for LLM Applications gives shared vocabulary here for prompt injection, sensitive information disclosure and excessive agency, which makes findings easier to triage against a framework the client’s development team may already use.
Testing to Consequence: Judge Findings by What the System Did
Agree a consequence threshold in writing before testing starts. It defines what counts as proof and what counts as too far. Without it, every severity argument during report review becomes a matter of taste.
The contrast is simple. A model that says it will email the customer list is a behaviour observation. A model that triggers a send, and the message leaves the mail queue to a tester-controlled address carrying a canary record, is an exploited vulnerability with a blast radius you can measure. One of those gets fixed. The other gets argued about.
Safe proof artefacts make that possible without causing harm: canary records seeded into the data set, benign marker files in the document store, test recipient addresses, scoped API keys that permit the call but limit its reach, and write operations aimed at a dedicated test project rather than production. Scoring should follow blast radius and reversibility, not the shock value of the transcript. A quiet cross-tenant read is worse than a lurid refusal-bypass, and the report should say so.
Handling Non-Determinism Without Losing Credibility
Non-determinism testing of AI systems is where most reports fall apart. A payload that works three times in ten is still a finding. It is a finding with a success rate, and the honest move is to state the rate rather than screenshot the lucky run.
Record the variables that actually change the outcome: model name and version, temperature or sampling settings where you control them, the system prompt revision in force, the retrieval corpus contents, the conversation history, and the date of the test. Fix a repetition protocol and stick to it. Ten attempts per payload, successes and failures logged, is enough to be credible without burning the engagement. The finding then reads “succeeded 4 of 10 attempts against model build X with system prompt revision 7, tested 15 September 2026″rather than “the model can be jailbroken”. The first survives retest. The second gets dismissed.
Plan the retest before the vendor ships an update, because they will. And note plainly in the report that a guardrail change which drops a payload’s success rate to zero has not closed the root cause. The model still reads instructions and data through the same channel. Filtering is a probability reducer, not a fix, however much that dents the demo.
Writing the Report So Engineering Can Act On It
Separate root cause from exploit path. The root cause section says the model cannot reliably distinguish instructions from the data it reads, and that this is inherent to how these systems work. The exploit path section gives the specific payload, source and tool chain. Keep them apart, because if the two are merged the development team fixes the wording and declares victory.
Primary recommendations should be structural: scope every tool credential down to least privilege, require human confirmation on irreversible or outbound actions, treat model output as untrusted input to whatever consumes it, and isolate retrieval per tenant and per user. Input and output filtering goes in as a partial layer, labelled so. Governance framing, where the client needs it, maps cleanly onto the NIST AI Risk Management Framework and ISO/IEC 42001.
The evidence package carries full transcripts, timestamps, model and version identifiers, tool call logs and the canary artefacts produced. One more thing: if any part of the engagement passes client content through a third-party AI service, say so, name the service, and exclude client data, credentials and personal data from it. UK GDPR obligations do not pause because the processing happens inside a model.
Where Automated Tooling Fits
Scanners earn their place on breadth. They run large payload libraries fast, they make good regression gates after a release, and they surface candidates worth manual follow-up. That is genuine value and it should be used.
Apply one test question to any tool before its output goes near a client report: is a given fact a real measurement or a model-generated assertion? If the tool measured it, the evidence exists. If the model stated it, you have a hypothesis. There is a great deal of noise in this space, and a fair amount of what is claimed for AI-assisted tooling is marketing.
Automation struggles exactly where the impact lives: multi-step tool chains, cross-tenant context, and verifying that an action actually completed in a downstream system. The sensible split is automation for coverage and regression, human testers for chaining and impact proof. The judgement about what a finding means stays firmly human. Further detail on the limits sits in this piece on what automated LLM red teaming misses.
The Short Version
Map permissions and inputs before sending a prompt. Treat the model’s authority as the union of every credential its tools hold. Work the attack classes in dependency order so injections can be chained into real actions. Judge findings by consequence, prove them with canaries, state attempt counts and model versions, and separate root cause from payload in the write-up. If the methodology produces those artefacts, the engagement stands up. For a structured walk-through of the engagement itself, see how this site approaches red teaming a large language model, and the five-phase field guide for the prompt-level detail that sits beneath this framework.
Frequently Asked Questions
What is an AI red teaming methodology, and how does it differ from a jailbreak test?
A methodology is the governing process for an engagement: surface mapping, permission enumeration, attack classes worked in order, consequence proof, and reproducible evidence. A jailbreak test is a single technique applied to the model in isolation. One produces findings an engineering team can fix; the other usually produces a transcript and an argument about severity.
How long does an AI red team engagement usually take?
It depends almost entirely on the number of tools and input channels, not on the model. A read-only chat assistant with no connectors is a short piece of work. An agent with several authenticated tools, a shared retrieval corpus and outbound actions needs time for setup, indirect injection staging and consequence verification, and the repetition protocol adds runtime on top. Scope by tool count and permission breadth.
How do you report a finding that only reproduces some of the time?
State the attempt count and the success rate, then pin down the conditions: model version, system prompt revision, retrieval contents, conversation state and test date. “Succeeded 4 of 10 attempts”is a credible finding. A single screenshot of a lucky run gets dismissed on retest and the issue stays open.
Can automated tools handle AI red teaming on their own?
No. They cover known payloads at breadth and make good regression gates, which is useful. They are weak on multi-step tool chains, cross-tenant context and confirming that an action actually landed in a downstream system, and those are the areas where the serious findings sit.
What should be in scope when red teaming an AI agent with tool access?
Every tool definition and the credential behind each one, every input source the agent reads including documents, tickets and fetched pages, the retrieval layer and its permission model, the output channels and whatever consumes them. Ask for write access to at least one ingested content source so indirect injection can be tested. Agree the consequence threshold and safe proof artefacts in writing first.