Two jobs hide behind the phrase AI penetration testing, and confusing them is the first mistake. One job is testing an AI system as the target: the chat interface, the retrieval store, the agent and everything it is allowed to call. The other is using models to help test ordinary web apps, APIs and infrastructure faster. Both are legitimate. They need different scopes, different evidence and different people in the room.
Most scoping documents blur them. The result is a statement of work that promises “AI security testing”, a tester who spends a fortnight writing jailbreaks, and an argument at report stage about why nobody enumerated what the model could actually do. This piece is a scoping guide, not a definition piece. It assumes you already know roughly what a large language model is and that someone has asked you to arrange a test by a date.
Two different jobs, one phrase
Job one: the AI system is the target
Here the thing under test is a product. A support assistant with retrieval over historic tickets. A coding agent with repository access. A document summariser fed by an ingestion pipeline. The interesting questions are about behaviour and permissions: what text can reach the model, what the model can do once it has been steered, and how far the damage travels.
Job two: AI assists a conventional test
Here the target is the same as it always was. A web app, a cloud tenancy, an internal network. The model reads, correlates, summarises and drafts. It does not decide whether a port is open or a parameter is injectable. Scope for this job looks like a normal penetration test scope with an extra section on tooling, data handling and evidence discipline. Name the model service being used, the data classes allowed anywhere near it, and the person who reviews model-drafted text before it reaches the client.
The decision test
Ask one question. Is the risk you are worried about the model’s behaviour, or the speed and coverage of testing you already do? If it is the former, you are scoping job one. If it is the latter, you are buying efficiency and should say so in plain words. Organisations that try to buy both under a single fixed-price line item end up with a report that is half curiosity and half reconnaissance notes. There is a great deal of noise in this space, and vague scopes are where it collects.
What belongs in scope when the AI system is the target
Scope the permissions before the prompts
This is the single most useful change you can make to an AI test scope. Enumerate what the model is allowed to do before anyone writes a single adversarial prompt. Every tool, every function call, every API credential, every database it can read and write, every mailbox it can send from, every ticket it can close, every refund it can issue.
Capture each one as a row with six fields: the tool name as it appears in configuration, the credential it uses, the scope of that credential, whether the call reads or writes, whether a human approves it before it fires, and what the worst outcome is if it fires with attacker-chosen arguments. A filled-in row reads something like: refund_order, billing service account, write access across the whole tenancy, write, no approval step, arbitrary credit to any customer account. Six fields per tool is enough to sort the list into the calls that need testing first and the ones that can wait.
Weak tests skip this. They produce a report full of clever bypasses and miss the fact that the support assistant can credit a customer account without a second pair of eyes. The bypass is the interesting bit for the tester. The refund authority is the interesting bit for the business.
List the trust boundaries
Write down everyone and everything that can put text in front of the model. Not just users typing into a box. Indirect paths are where the serious findings usually are: uploaded documents, inbound email, support tickets, calendar invites, web pages the agent fetches, code comments, issue descriptions, PDF metadata, shared memory between sessions. Order that list by whether the text reaches the context window without a human having read it first, because those channels are the ones worth testing on day one. If a channel can carry attacker-controlled text into the context window, it is in scope or it is explicitly excluded in writing. No middle ground.
Separate sensitive output from sensitive action
Name both in the scope, because they need different tests and carry different severity. Sensitive output is disclosure: system prompt contents, another tenant’s data, records the user was never entitled to see. Sensitive action is a change in the world: a payment, a commit, an email, a deletion, a permission grant. A test that only looks for forbidden words will find the first class and miss the second entirely.
The omissions that recur
Document ingestion pipelines get left out because they feel like plumbing. Ticket and email feeds get left out because nobody thinks of them as user input. Shared or persistent memory gets left out because it was added in a sprint after the architecture diagram was drawn. Integrations bolted on after launch are almost never in the diagram at all. Ask for the list of tools as it exists in configuration today, not as it exists in the design document. Once that list exists, the evidence questions that expose weak tooling before purchase are worth asking of any vendor you are considering.
The test classes that produce findings worth acting on
A workable sequence, roughly in the order it pays to run: instruction bypass, indirect injection, data disclosure, tool abuse, excessive agency. The first two establish that you can steer the model. The last three establish what that steering is worth. In practice that means five concrete probes: a direct instruction-override attempt typed into the chat box; the same payload hidden in an uploaded PDF and in the body of a support ticket; a request for the system prompt and for a record belonging to another tenant; a tool call carrying arguments the user should not be able to set; and a write action attempted with no approval step in front of it. The OWASP Top 10 for LLM Applications is a reasonable spine for the class list, and MITRE ATLAS is useful when you need technique names for adversarial machine learning that a defender will recognise. For governance framing, the NIST AI Risk Management Framework and its generative AI profile give you language that maps onto board reporting.
Test to consequence
Judge a finding by what the system did, not by whether the model said something it was told not to say. A model reciting a forbidden sentence in a sandbox is a curiosity. The same model calling a tool with attacker-chosen arguments is an incident.
The example to hold in your head: a support assistant with retrieval over historic tickets and a refund tool. An attacker files a ticket whose body contains instructions. Weeks later a genuine customer query pulls that ticket in as context. The model reads the instructions and issues a refund. The finding is the refund. The text is just the delivery mechanism.
Prompt injection has no clean fix, so stop asking for one
A model cannot reliably separate the instructions it was given from the data it reads. Both arrive as text in the same window. That is the whole vulnerability, and it follows from the design rather than from a coding error. Scope documents that ask a tester to confirm prompt injection has been remediated are asking for something no provider can honestly deliver.
Ask for something else instead: evidence that a successful injection stays survivable. Narrow permissions, human approval on write actions, allowlists on tool arguments, output validation outside the model, logging that lets you see what happened. Concretely, for the refund example: the tool can only target the order ID attached to the current conversation, it is capped at a value the business is willing to lose without review, anything above that cap queues for a human, and every call is logged server-side with the arguments it was given. Recommendations shaped this way survive the next model update. Filter-tuning recommendations do not. If you want the longer argument, this site has a full treatment of why prompt injection defines LLM security.
Look for the confused deputy
The model is fooled, but the damage comes from its permissions. A coding agent with read and write access to a repository and an issue tracker is the cleanest illustration. An untrusted issue description steers a commit. Nothing was hacked in the traditional sense. The agent did exactly what it was asked, using authority it legitimately held.
The fix is permission scoping and human approval on writes, not better filtering. That distinction belongs in the report, and it belongs in the scope as a question you intend to answer. Practical detail on constraining tool access sits in the guide to securing AI agents and their tool use.
Scoping AI-assisted testing of conventional targets
Where models genuinely help
Reading and synthesis. Sifting job adverts, documentation, commit history and certificate transparency logs during reconnaissance. Correlating findings across tools. Drafting the tedious half of a report so the tester can spend the time on the technical half. None of this replaces the tester. It removes the drudgery around the edges, which matters, because drudgery is where fatigue sets in and things get missed. There is a fuller account of the useful cases in the notes on using large language models for reconnaissance.
Keep the model out of verification
The rule is simple and absolute: a model may organise and explain information, but the information itself must come from a tool that actually queried the target. Never let the model be the source of a fact. Break that rule and you will publish hallucinated evidence under your own name, with your own logo on the cover page.
The question for any tool vendor
Does this fact come from a measurement or from a model’s guess? Ask it about every field in the output. A confirmed SQL injection should trace back to a request, a response and a differential. A CVE claim should trace back to a version string the tool read, not a version the model inferred from a banner that looked familiar. A great deal of what is claimed for AI in security is marketing, and this one question separates the tools that measure from the tools that narrate. There is more on that split in this assessment of what works and what is hype in AI-assisted pentest tools.
What to require in the statement of work
Evidence trails for every finding. Reproducible steps a defender can run. Named tooling per finding, so the reader knows whether a result came from a scanner, a manual request or a model summary. A clause that model-generated content is labelled so. Ask for the same header block on every AI finding: test class, entry channel, tool called and arguments used, attempts and successes, model name and version, and the server-side log reference. That last one sounds bureaucratic until the first time a client asks where a number came from.
Non-determinism, evidence and severity
One successful prompt is an anecdote. Run it again. Handle non-determinism in the method rather than apologising for it in the report.
Practical approach: repeat each probe across multiple runs and report the observed success rate rather than a single screenshot. Ten runs per probe is a sensible floor, twenty where the consequence is a write action, and the report should state the attempts as well as the successes. Three successes out of twenty attempts is a real finding with a real rate attached. It also gives the defender something to measure after they change something.
Record enough state that the behaviour can be reproduced. Model name and version. System prompt state at the time. Temperature and other sampling settings if you can get them. Tools available to the model. The exact input, including any retrieved context. The exact action the system took, taken from server-side logs rather than from the chat transcript.
Write severity around consequence and blast radius, not around the novelty of the bypass. A workable rubric: an unapproved write to production data is high or critical, cross-tenant disclosure is high, and disclosure of the system prompt on its own is low unless it carries credentials or endpoint detail. An elegant multi-turn jailbreak against a read-only summariser is low severity. A crude one-line injection against an agent that can write to production is high, however much that dents the demo.
Findings sometimes stop reproducing after a vendor updates the model. Do not quietly drop them. Report them with the version they were observed against, note that the behaviour was not reproducible after the update, and keep the structural recommendation. The permission that made the finding dangerous is still there.
Third-party models, engagement data and trust
Third-party model use is a data-handling decision as much as a technical one. Passing engagement notes, configuration files, scan output or client personal data into a hosted model raises retention, logging and sub-processor questions. Settle them in the engagement letter, not mid-test.
UK GDPR and the Data Protection Act 2018 apply as soon as engagement data containing personal data reaches a hosted provider. The questions to answer before anything is sent: what is retained and for how long, whether inputs are used for training and how to opt out, where processing happens, which sub-processors are involved, and who the controller and processor are for this specific flow. The ICO’s guidance for organisations is the reference point for the wider obligations.
Controls that work in practice: redact identifiers before anything leaves your environment, use synthetic data for tooling experiments, run local models for sensitive material, and maintain an approved-tool list that names which services are permitted for which data classes. Redaction means the specifics that identify people and estates: client and subsidiary names, staff names and email addresses, internal hostnames, IP ranges, ticket and account numbers, and any credential or token that appears in captured output. Then put a clause in the engagement letter telling the client which parts of the work involve external models. Clients rarely object. They object to finding out afterwards.
Questions to settle before you sign off the scope
For a vendor: which test classes are included by name, what evidence is delivered per finding, how non-determinism is handled, and whether findings are traced to server-side logs or to chat transcripts.
For an autonomous platform: what does it validate against the target itself, and what does it assert from a model? Who reviews the output before it reaches the client? If the answer to the second question is nobody, you are buying a document, not a test.
For your own team: which permissions can be cut now, regardless of what the test finds? Write-access an agent has never used. Tool calls without approval gates. Credentials scoped to a whole tenancy because that was easier. Cutting those before the test makes the test more useful, because the remaining findings are the ones you could not guess.
A short readiness list before day one: a non-production environment that behaves like production, logging that captures tool calls and arguments, rate limits raised or documented so the tester is not measuring throttling, model version pinned for the duration, and an agreed rollback for any state the testing changes.
The Short Version
Decide first whether the AI is the target or the assistant, and write two scopes if you need both. Enumerate permissions and tool access before anyone writes a prompt. Judge findings by what the system did, not by what it said. Do not ask anyone to prove prompt injection is fixed; ask for evidence that a successful injection is survivable. Run probes repeatedly and report a rate. Keep language models on reading and synthesis and off verification. Settle the data-handling questions about hosted models in the engagement letter. Further analysis and how-to material on AI penetration testing and offensive security sits across this site, and the judgement in all of it stays firmly human.
Frequently Asked Questions
What does AI penetration testing actually cover?
It covers two distinct pieces of work. Testing an AI system as the target means assessing the chat interface, retrieval stores, agent tool calls and permissions for instruction bypass, indirect injection, data disclosure, tool abuse and excessive agency. Using AI to assist a conventional test means models help with reading, correlation and drafting while real tools do the measuring. Scope them separately.
How is AI penetration testing different from a standard web application pentest?
The application layer still needs the usual testing, so the standard work does not go away. What is added is a non-deterministic component that reads untrusted text and can take actions with real authority. That shifts the focus from input validation flaws to trust boundaries, permission scope and what happens after the model is successfully steered.
Can autonomous AI tools replace a penetration tester?
No. They can expand coverage, sift output and handle repetitive checks, which is genuinely useful. They cannot decide what matters in a particular business, chain findings with judgement, or tell you that a technically minor bug is severe because of where it sits. Ask any platform what it verifies against the target and what it merely asserts.
How do you report a finding when the model’s behaviour is not repeatable?
Run the probe many times, ten as a floor, and report the successes against the attempts rather than a single screenshot. Pin the model name and version, the system prompt state, the tools available and the exact action taken, drawn from server-side logs. If the behaviour stops reproducing after a vendor update, keep the finding, note the version it was seen against, and keep the permission-scoping recommendation.
Is it safe to send engagement data to a third-party AI service?
Only after the data-handling questions are answered. Check retention periods, training opt-outs, processing locations and sub-processor lists, and confirm the UK GDPR and Data Protection Act 2018 position for the data in question. Redact identifiers such as client names, staff email addresses, internal hostnames and IP ranges, prefer synthetic data for experiments, use local models for sensitive material, and tell the client in writing which parts of the work involve external models.