A checklist is only useful if it tells you what to look at, and most of the LLM penetration testing checklists in circulation point the tester at the wrong thing. They point at the chat window. Prompt injection, data leakage, excessive agency, jailbreaks: the categories are right, but the pass/fail criterion is almost always some version of “did the model say something it should not have said”. That is a measurement of text. It is not a measurement of harm.
The checklist below is organised around consequences instead. Every item has a pass criterion written in terms of what the system did: a tool call fired, a record changed, a document was retrieved outside its entitlement, data crossed a boundary. Findings written that way do not get argued away in the debrief.
Why most LLM testing checklists stop one step too early
A jailbroken sentence is a lead. It tells you the instruction hierarchy is soft, which is interesting, but on its own it is a model quirk and it will be treated as one. The vendor will say the model was role-playing. The application owner will say nothing happened. Both are, in the narrow sense, correct.
What turns that lead into a finding is the next hop. The injected instruction reached an orchestration layer that acted on it. The agent called a tool. A ticket field changed. An email left the tenant. A chunk of an HR document appeared in a standard-tier user’s answer. The artefact is the audit log entry, not the transcript line.
So set the rule for the whole exercise before you write a single payload: an item only passes or fails on an observable consequence. If a check cannot be written that way, it is reconnaissance, not a test case. That single rule quietly removes about a third of the usual checklist and replaces it with work that matters.
Name the categories anyway, because findings have to land with people who maintain risk registers. Map each check to the OWASP Top 10 for LLM Applications (prompt injection, sensitive information disclosure, excessive agency, vector and embedding weaknesses are the four you will use most) and give the technique a name from MITRE ATLAS. Governance readers need the taxonomy. Engineers need the consequence. Give them both in the same finding.
Before the first payload: map permissions, inputs and data flows
Testing an LLM application without this step produces a list of clever prompts and no idea whether any of them matter. Spend the first part of the window on paper.
- Every input the model reads. Not just the user prompt. The system prompt, retrieved documents, tool responses, file uploads, email bodies, ticket text and attachments, calendar invites, scraped web pages, code comments. Anything that reaches the context window is an input, whoever wrote it.
- Every tool, function and connector, with its exact credentials and scopes. Write down the identity each call runs as. Pay attention to anything inherited from a hosting service account, because that is where over-provisioning hides. A read-only API key on paper often turns out to be a tenant-wide token in practice.
- The trust boundary, drawn properly. Mark where untrusted text enters the context, and mark everything the model can reach once it is in there. Those two lines together are the attack surface. The rest is detail.
- The baseline. Model name and version, system prompt (keep a hash of it), temperature and other sampling settings, guardrail layers and their versions, retrieval index state and when it was last built. Findings are only reproducible against a fixed baseline, and an LLM baseline moves under you unless you record it.
Scoping and access arrangements for this kind of engagement are covered separately in the guide to scoping an AI penetration test; get the logging access agreed there, because without tool call logs most of this checklist degrades back into reading chat output.
Part one: instruction bypass and guardrail limits
Run these to establish how soft the instruction hierarchy is, then treat the results as inputs to parts two and three rather than as findings in their own right.
- Direct override and role reassignment. Ignore-previous-instructions variants, new persona, fake system turn, fake developer message.
- Encoding and language switching. Base64, homoglyphs, zero-width characters, a payload in Welsh or Japanese, instructions split across turns and reassembled.
- Multi-turn drift. Twenty benign turns, then the ask. Guardrails tuned on single prompts often lose the thread.
- Persistence. If the product has memory, saved instructions or a user profile the model reads, plant an instruction there and come back in a fresh session. Persistence changes severity more than any single payload.
- System prompt and configuration disclosure. Direct extraction, but also partial leakage through error text, verbose tool arguments and echoed function schemas. A leaked tool schema tells an attacker exactly which verbs to aim for.
- The guardrail as a component. Test it on its own terms. What does it block, what does it merely log, and what happens downstream when it fails open? A classifier that drops the request but lets the already-issued tool call complete is a finding about the orchestration layer, not about the classifier.
Write the result carefully. Not “the model complied with a jailbreak”, but “a user-supplied instruction was accepted as a privileged instruction by the orchestration layer, which then executed the resulting tool call without confirmation”. The first sentence invites a debate about model behaviour. The second describes a control failure. Background on why this class exists at all is set out in this site’s piece on prompt injection as the defining LLM vulnerability.
Part two: indirect injection through content the model ingests
This is where the serious findings usually are, and it is the part that public jailbreak lists never reach. Nobody has to attack the chat interface. They only have to write something the system will read later.
- Plant instructions in real sources. A knowledge base page, an uploaded PDF, a ticket attachment, a calendar invite, a code comment, a page the agent is allowed to fetch. White text, a footnote, an HTML comment. Then let the normal workflow pick it up.
- Index poisoning. Can an unprivileged user get content into the retrieval corpus? If a support agent’s notes or a public-facing form feeds the index, the answer is usually yes.
- Entitlement mismatch in retrieval. Query as a standard-tier user and check whether chunks arrive from documents that user cannot open directly. The common cause is a shared document store whose access control lists were never mirrored into the vector database. Evidence is the retrieved chunk alongside the source document’s ACL. A model refusal proves nothing; the chunk was already out.
- Cross-tenant chunk leakage. Same index, different customers, metadata filtering applied at query time only. Test the filter, not the prompt that depends on it.
Take the support assistant that retrieves from an internal knowledge base and holds tools to read tickets, update ticket fields and send email. An attacker raises a ticket with an attachment containing instructions. A human agent later asks the assistant to summarise that ticket. The assistant reads the attachment, follows it, and emails the account notes to an external address using its own mailbox permissions. The finding is the sent message and the corresponding audit log entry. The chat text is supporting evidence.
State the root cause in the report in one sentence, because remediation depends on it: the model cannot separate the instructions it is given from the data it reads. Delimiters, input filters and guardrail models reduce the hit rate. They do not close the class. Written that way, the recommendation stops being “improve the prompt filter”and becomes “scope the tokens and gate the irreversible actions”.
Part three: disclosure, tool abuse and excessive agency
Disclosure. Cross-session bleed and cross-tenant bleed. Secrets sitting in the context window (API keys pasted into a system prompt happen more often than anyone admits). Verbose tool output returning whole records when one field was needed. Prompts and retrieved content written into application logs, traces and observability platforms that a wider group can read.
Tool abuse. Injected arguments. Unsafe parameter values (a wildcard where an identifier belongs, a date range of “all”). Chained calls where the output of one becomes the input of the next without re-authorisation. Destructive verbs: delete, refund, deactivate, merge. And outbound requests as an exfiltration channel, including the quiet one, an image or link rendered in the response with the stolen data in its query string.
Blast radius, scored per tool. For each tool, record the identity it runs as, the write operations it permits, the records it can reach and the network paths it opens. That table is the real severity model for an agent, because the question is not how clever the payload was. The question is what an attacker inherits the moment the model is fooled. An agent acting on an attacker’s instruction with its own legitimate credentials is the confused deputy pattern, and it is the reason least privilege, scoped tokens and human confirmation earn their keep. The equivalent exercise for one protocol is worked through in this site’s MCP server testing checklist.
Excessive agency. List the actions available with no confirmation step, the spend and rate limits (and whether they are enforced server-side), the irreversible operations, and the end-to-end reach of a single compromised turn. If one turn can send mail, change a record and call an external API, say so plainly and show the chain.
Proving it: repeat runs, non-determinism and findings that survive review
Non-determinism in testing AI systems is a procedural problem, not an excuse. Handle it the same way every time.
Fix the baseline. Run each confirmed technique a set number of times, twenty is a workable default, and record the success rate as a fraction. A 5-in-20 reproduction is a real finding; sampling variance does not make an email that left the tenant unreal. An unrecorded one-off is not actionable, which is a different problem and an avoidable one.
Every finding needs an evidence pack: the full transcript, the tool call logs with timestamps, model version and system prompt hash, temperature, the resulting state change (the sent item, the modified field, the retrieved chunk with its source ACL) and reproduction steps someone else can follow. Grade severity on consequence and reachability, never on how alarming the output reads.
Retest with the same protocol. Twenty runs against the same baseline, success rate recorded. If a vendor says the issue is fixed and the retest was three attempts on a newer model version, the fix is unverified. Say that in the report. The judgement about whether something is closed stays firmly human, however much that dents the demo.
Checklist mistakes that waste a test window
- Treating a public jailbreak list as coverage. Hundreds of prompts, zero retrieval tests, zero tool permission review. Volume, not depth. The limits of that approach are set out in this site’s analysis of what automated LLM red teaming misses.
- Testing a staging build with the guardrails switched off, or an index with twelve lorem ipsum documents in it. Neither result transfers to production. Insist on realistic data or state the gap in the report.
- Accepting AI-assisted tool output as evidence. Ask one question of anything a model hands you during the engagement: did that come from a measurement or from the model asserting it? Use models for reading logs, summarising and drafting candidate payloads. Keep them out of verification.
- Pasting client prompts, logs or retrieved documents into third-party services without checking the data-handling position. UK GDPR obligations on processors and international transfers apply to engagement material like any other personal data, and some providers retain prompts for training unless that is turned off. Check before, not after.
- Leaving the design question unanswered. Assume the model will be fooled eventually, then test whether the permission model makes that survivable. If the honest answer is no, that is the headline finding, and it is a better one than any individual payload.
How these checks fit into a full engagement, from scope through to proof, is covered in the service page on red-teaming a large language model, and the agent-side defences worth recommending are gathered in securing AI agents and their tool use.
The Short Version
Write every item in your LLM penetration testing checklist with a consequence as its pass criterion. Map inputs, tools, credentials and the trust boundary before the first payload, and record the baseline so findings reproduce. Treat instruction bypass as a lead, indirect injection through ingested content as the main event, and per-tool blast radius as the severity model. Run twenty attempts, record the success rate, keep the transcripts and the tool logs, and retest with the same protocol. State the root cause once: the model cannot tell instructions from data, so the fix lives in permissions and confirmation gates, not in better filters.
Frequently Asked Questions
What should an LLM penetration testing checklist actually cover?
Four blocks: the pre-test map (inputs, tools, credentials, trust boundary, baseline), instruction bypass and guardrail limits, indirect injection through retrieved and ingested content, and then disclosure, tool abuse and excessive agency. Each item needs a pass criterion expressed as an observable consequence. Map the findings to OWASP Top 10 for LLM Applications categories and name techniques using MITRE ATLAS so they slot into an existing risk register.
How is testing an LLM application different from a standard web application pentest?
The input set is wider, because any document, ticket or web page the system reads is an input, and the same payload does not always produce the same result. Classic web testing still applies underneath (authentication, authorisation, injection into the surrounding APIs) and should not be dropped. What is added is non-determinism, a retrieval layer with its own permission model, and tools that act with the application’s credentials rather than the user’s.
How many times should each prompt injection attempt be run before it counts as a finding?
Run each confirmed technique a fixed number of times against a fixed baseline and report the fraction; twenty attempts is a practical default. A 3-in-20 or 5-in-20 reproduction is a genuine finding if the consequence is real, as long as the success rate and the baseline are recorded. Use the same number of runs at retest, otherwise the claimed fix is unproven.
Can automated scanners replace a manual LLM testing checklist?
No. Automated tools are good at breadth, generating payload variants and flagging where the model emitted something unexpected, and that saves real time. They are poor at the part that produces serious findings: understanding which tool runs under which credential, planting content in the sources the system genuinely reads, and judging whether a state change matters to the business. Use them for coverage and keep the judgement human. Before you buy on the strength of a demo, put the tool through the six questions that sort genuine capability from marketing claims.
Is it safe to send engagement data to third-party AI services during a test?
Only after a data-handling review. Prompts, logs and retrieved documents from a live system frequently contain personal data, so UK GDPR duties around processors and international transfers apply, and some providers retain inputs for training unless that setting is changed. Check the current terms for the specific service and tier, get the client’s written agreement, and keep evidence and verification work on infrastructure you control.