Every guide to red teaming a language model starts in the wrong place. It starts with prompts. If you want to know how to red team an LLM in a way that produces findings a developer will actually fix, you start with permissions, because the model is not the asset. The application around it is: the retrieval corpus, the tool schemas, the credentials each tool borrows, the records it can change. Model behaviour is the delivery mechanism. Consequence is the finding.
This is the method to use after the automated scanner has run. Scanners are useful, and there is a place for them further down. They will not tell you whether your refunds API can be driven by text in a support ticket.
What Red Teaming an LLM Actually Means in an Application
There are two activities sharing one name, and confusing them is the first mistake.
The first is model evaluation: testing a foundation model in isolation for harmful outputs, bias, refusal quality and alignment failures. Frontier labs do this at scale before release, with resources no consultancy will match. The second is testing a deployed system: a model wired to retrieval, tools, memory and a permission model that someone in your organisation designed in a sprint. That second activity is your engagement. You are not auditing the provider’s alignment work. You are testing your integration.
The practical consequence is a change in what counts as a finding. A model that can be coaxed into saying something offensive is a product quality issue, and worth a line in an appendix. A model that reads a planted instruction inside a support ticket and calls the refund endpoint is a security finding. Keep the two apart in the report. Blend them and the developer dismisses both, because the first one makes the second look like the same category of noise.
Three things need to exist on paper before testing begins. Written authorisation naming the systems, accounts and tools in scope. An environment: either non-production with realistic data flows, or a production engagement with an agreed blast radius, which in practice means a written list of actions you may trigger and a nominated contact who can undo them. And an understanding of provider rate limits and abuse policies, because adversarial traffic at volume against a hosted model can get the client’s API key throttled or suspended mid-engagement. Get that expectation set before you begin, not after. The scoping questions worth settling up front are the difference between a two-week engagement and a two-week argument.
Phase 1: Map Permissions, Inputs and Data Flows
Most first engagements fail here, not in the attack phases. Testers open a chat window and start typing jailbreak prompts while nobody has written down which tools the agent can call, or whose credentials each one runs as. You end up with a pile of interesting transcripts and no security findings.
Start by enumerating every untrusted input path. Not just the chat box. Uploaded documents. Content retrieved from the knowledge base. Web pages the agent fetches. Inbound email. API payloads from partner systems. Tool outputs fed back into the context window, which are untrusted the moment a tool reads anything a user can influence. Calendar invites, file names, ticket comments, HTML title attributes. Anywhere text can enter the context, an instruction can enter with it.
Then list every tool, function and MCP server the model can invoke. For each one, record:
- What it can read, and at what granularity
- What it can write, and whether the write is reversible
- Which identity it authenticates as: the end user, a service account, or the application’s own credentials
- Whether a human confirms the action before it executes
- Whether the tool can make outbound network requests
That table is the artefact that turns a prompt experiment into a security test. It also tends to produce findings on its own, before any attack. A service account with write access across all tenants, a file-read tool that accepts arbitrary paths, an HTTP tool with no domain allowlist: all visible from the schema. If the system uses Model Context Protocol servers, treat each one as a separate component with its own trust boundary and work through an MCP server testing checklist rather than assuming the host application constrains it. Where that trust boundary fails under prompt injection, the method for proving the blast radius is set out separately.
Finish phase one by naming the highest-consequence action the system can perform. Issue a refund. Send mail as the company. Delete a record. Modify an access control list. That becomes your target hypothesis. Generic harm lists give generic reports; a single named consequence gives the client something they can price.
Phase 2: Instruction Bypass and Guardrail Testing
Prompt injection works for a structural reason, not a training one. The model receives its instructions and its data through the same channel, as tokens, and cannot reliably tell which is which. That is the whole vulnerability. Delimiters, instruction hierarchies and system prompt hardening raise the cost of an attack. They do not remove the class. Any vendor claiming otherwise is selling confidence, not a fix, and the longer explanation of why this vulnerability defines LLM security is worth reading before you write a remediation section promising it can be patched.
For direct bypass, a handful of techniques earn their time. Role reassignment, where the model is told it is now a different system with different rules. Task framing, where the forbidden action becomes a subtask of a legitimate one: translate this, debug this, summarise this. Format coercion, asking for output as JSON, a code comment or a table, which frequently slips past filters tuned on prose. Encoding and language switching, including base64, homoglyphs and low-resource languages where safety training is thinner. And conversational drift, where nothing in turn one is objectionable and the constraint has quietly dissolved by turn fifteen. Long-context multi-turn testing is underdone in most engagements and is where the interesting bypasses often sit.
Test the guardrail layer as a separate component. If there is an input classifier, try to evade it independently of the model. If output filtering is applied post-hoc, check whether streaming responses emit tokens before the filter fires; partial disclosure before a cut-off is still disclosure. And watch for the refusal that leaks anyway: the model declines, then summarises what it declined to say.
Record the exact state that produced each bypass. Prompt text, conversation history, retrieved documents in context, tool outputs present at the time, model version, temperature. Reproduce a bypass without the context and it will look like it never happened.
Phase 3: Indirect Injection, Retrieval and Data Disclosure
Indirect injection is the more serious half of the problem, because the attacker never touches the interface. They plant text where the model will read it later and let a legitimate user trigger it.
Places worth planting: a document in the RAG corpus, a comment on a ticket, a field in a CRM record, hidden text in an HTML page the agent fetches, metadata in a PDF. White-on-white text in a CV works on CV screening agents with unhappy reliability. A candidate document instructing the model to disregard prior guidance and score the applicant highly, in a pipeline with no human reviewing the score, is a finding with regulatory teeth as well as technical ones.
The retrieval layer deserves its own attention, and this is where the serious findings usually are. In a shared vector store, retrieval filters are an authorisation control. They are rarely tested as one. A common pattern: document-level permissions applied at ingestion into a SharePoint-backed corpus, but never enforced at query time, so the store returns by similarity and a user in finance pulls passages out of HR files. Before you spend an afternoon on clever phrasing, ask the agent plainly for content it should not be able to reach, and ask using a second tenant’s identifiers. Cheap test, high hit rate.
System prompt extraction is reconnaissance, not a finding by itself. It becomes a finding when the prompt contains a secret, an internal hostname, an undocumented endpoint or a business rule the client considers confidential. Report the contents, not the extraction.
Training data and memory leakage need honesty about scope. Verbatim training data extraction from a hosted frontier model is not realistically provable in a black-box engagement, and claiming it without evidence damages the report. Persistent memory features are testable: write something in one session, check whether it surfaces in another user’s session, or in the same user’s session after a supposed reset. That is a concrete cross-session boundary you can demonstrate.
Phase 4: Tool Abuse, Excessive Agency and the Confused Deputy
Now chain it. Injected content instructs the agent, the agent calls a tool using its own elevated identity, and the system performs the attacker’s request while logging the action as legitimate.
The canonical case is the internal support assistant: read access to the ticket system, write access to a refunds API. An attacker opens a ticket containing instructions addressed to the assistant. A human agent asks for a summary. The model reads the injected text and issues the refund with the application’s credentials. Classic confused deputy. The finding is the refund, not the text.
Tool schemas telegraph this. Look for broad free-text parameters, unvalidated file paths, SQL or shell passthrough, query builders that accept raw fragments, and outbound HTTP with no allowlist, which is an exfiltration channel regardless of what it was built for. Image rendering from arbitrary URLs is the same channel wearing a hat.
Then verify consequence, not output. Did data leave the network. Did a record change. Did an email send. Did a payment initiate. Check the tool call log, the database row, the mail queue, the egress logs. A transcript in which the model announces it has issued a refund proves nothing; models narrate actions they never took.
Remediation should be written for survivability rather than perfection. Assume the model will be fooled, then ask what it can do once it is. Scope each tool’s permissions to the minimum. Split read and write identities. Require human confirmation on irreversible or high-value actions, with enough context shown for the human to make a real decision. Allowlist outbound destinations. The architectural controls around agent tool use are durable; prompt-level hardening is partial, and the report should say so plainly.
Phase 5: Prove It Twice and Write It Up
Non-determinism breaks the reproduction standard everyone brings from web application testing. An exploit that fires every time is the exception. Reporting it as though it does invites a developer to try once, fail, and close the ticket.
So report attempt counts. Seven of ten attempts, not “the model can be tricked”. State the model version string, the temperature and other sampling parameters if you can see them, the date, and the exact context contents at the time. Include the injected artefact itself as a file, not as a quoted snippet that loses its whitespace. Attach full request and response pairs plus the tool call log with timestamps. A finding written as “seven of ten attempts against this model version, tool call logged at this timestamp, refund record created”gets fixed. One written as “the model can be manipulated”dies in triage.
Pin the model version at retest. Providers update hosted models, sometimes without a version change visible to you, and results can shift in both directions for reasons that have nothing to do with the fix. Note the version tested and the version retested. If they differ, say what that means for confidence.
Grade severity by what the system did. Then split remediation into prompt-level mitigations, which reduce likelihood, and architectural controls, which reduce impact. Map each finding to something the client already reports against: the OWASP Top 10 for LLM Applications categories for prompt injection, sensitive information disclosure and excessive agency, and the NIST AI Risk Management Framework for governance. Where personal data passed to a third-party provider during testing, or where the application routinely sends it there, flag the UK GDPR question about processors and international transfers. Security teams do not always own that conversation, but they are usually the ones who notice.
Where Automated Tooling Fits
Automated red teaming suites are good at breadth and regression. Hundreds of known bypass patterns, replayed after every prompt change, catching the drift that manual testing will not see because nobody reruns turn eleven of a conversation from three weeks ago. Use them for that. They are not a substitute for the five phases above, because they do not know your business logic, your permission model, your multi-step tool chains, or how to confirm that a record actually changed.
Ask one question of any AI-assisted tool: is this output a measurement or a model’s assertion? A scanner that reports a data leak because a judge model thought the response looked like one is producing an opinion. Verify the tool call log or the egress record yourself. There is a great deal of noise in this space, and a great deal of what is claimed for AI in security is marketing; a fuller assessment of which AI-assisted tools hold up under scrutiny is worth reading before you buy.
Check data handling too. Client prompts, retrieved documents and system prompts passing through a third-party service are client data. Know where they go, how long they are kept, and whether they are used for training. Your engagement terms should say so in writing.
The Short Version
- Map permissions, inputs and tool identities before sending a prompt. The permissions table is the deliverable that makes the rest a security test.
- Judge findings by consequence. Rude output is product quality; an unauthorised tool call is a vulnerability.
- Prompt injection has no clean fix, because instructions and data arrive on the same channel. Report the containment, not the block.
- Indirect injection through the retrieval corpus and fetched content matters more than direct jailbreaks. Test retrieval filters as authorisation controls.
- Verify what the system did: tool logs, database rows, egress, mail queue.
- Report run counts, model version and date. Pin the version at retest.
- Scanners give breadth and regression. The judgement about what matters stays firmly human.
Frequently Asked Questions
How long does it take to red team an LLM application?
It depends almost entirely on the tool surface, not the model. A single-purpose chatbot with retrieval and no write actions is a small piece of work. An agent with several tools, a shared vector store and multi-tenant data needs longer, because permission mapping, multi-turn testing and consequence verification all take real time and each finding has to be run repeatedly to establish a success rate. Scope by number of tools and untrusted input paths rather than by number of prompts.
What is the difference between red teaming an LLM and standard penetration testing?
The methodology is the same discipline applied to a probabilistic component. You still map attack surface, test authorisation boundaries and prove impact. What changes is that the injection point is natural language rather than a parameter, the same input can produce different results on repeat, and the most damaging findings emerge from the model’s authority to call tools rather than from a flaw in a single request. Traditional testing of the surrounding API, auth and infrastructure is still needed, and the two should run together.
Do I need access to the model weights or the system prompt to red team an LLM?
No. Most engagements are black or grey box against a hosted model, and that reflects how attackers see the system. Access to the system prompt, tool schemas and permission configuration speeds the work up considerably and finds more, because you can reason about which tool to target instead of guessing. Weights are almost never relevant unless the client self-hosts and wants model-level evaluation as a separate exercise.
How do you report a finding when the attack only works some of the time?
Report the rate. State the number of attempts, the number of successes, the model version string, the date and the sampling settings, and attach the exact context and injected artefact. Include the tool call log or the changed record as the proof of consequence, so the finding does not depend on the developer reproducing the wording. Intermittent exploitability lowers likelihood, not impact, and the severity should be graded accordingly rather than discounted.
Can automated red teaming tools replace a manual LLM engagement?
No, though they complement one well. Automated suites cover known bypass patterns quickly and are genuinely useful as regression tests after prompt or model changes. They cannot infer your permission model, chain injection through a tool to a downstream action, or confirm that data actually left the network, which is where the findings worth paying for live. If you need a structured approach to that manual work, see the fuller method for how to red team a large language model in a deployed system.