Confused Deputy AI Agent: 6 Tests That Prove It

Confused Deputy AI Agent: 6 Tests That Prove It

Most confused deputy write-ups you will find are about AWS. Set aws:SourceArn, set aws:SourceAccount, close the ticket. That advice is correct and almost entirely beside the point when the deputy is an LLM agent, because the condition keys constrain which service may invoke a role, and say nothing about whether the agent’s tool call reflected the entitlements of the person who asked for it.

A confused deputy AI agent is a privileged intermediary that takes instructions from text it read and executes them with its own authority rather than the requester’s. Prompt injection is how the instruction arrives. Confused deputy is why it matters. Confusing the two is the first mistake, and it produces findings that die in the debrief: a clever jailbreak transcript, no identity in a log, no state change, nothing a developer will fix.

What follows is the authority-boundary test. Six of them, each with what to plant, which identity to watch, and what a clean pass actually looks like.

What makes an AI agent a confused deputy

Norm Hardy named the pattern in 1988: a compiler with permission to write to a billing file is handed an output path by an unprivileged caller, writes there obligingly, and destroys the file. The compiler was not compromised. It was obedient, privileged, and unable to tell whose authority it was spending.

LLM agents are the sharpest version of this yet, because instructions and data travel down one channel. A ticket comment, a README, a row in a CRM, an HTTP response header: all of it reaches the model as text, and the model has no reliable way to mark one span as content and another as command. Any text the agent reads can become a request. That is the whole vulnerability.

The model is not the asset. Remove the tools and it is a chatbot that occasionally says rude things. Give it a service account with read access to every customer record, a CI token, a refund API, and the same chatbot becomes a financial control with no concept of who it is acting for.

Cloud-side controls do real work in their own lane. The AWS guidance on the cross-service confused deputy problem stops service X from being tricked into assuming your role on behalf of a stranger. It does not know whether the agent that just called your internal API was answering user A or quietly serving an instruction planted by user B. The decision layer sits above it, untouched.

Map the agent’s authority before you test anything

Permission mapping is not paperwork you do to look thorough. Without it you cannot tell a confused deputy from a normal authorisation bug, and you will not know which tests are safe to run twice.

Write down every credential the agent holds: service accounts, OAuth tokens, database roles, MCP server connections, shell or CI access, outbound email. Then mark each tool call with the identity the backend will actually see. Some calls carry a per-user token. Some run as the agent itself. The gap between those two columns is your confused deputy surface, and it is usually larger than the architecture diagram implies.

Record data breadth next. Not “can it read the CRM”but “for user A’s session, which rows for user B are inside the retrieval corpus or reachable by one tool call”. Finish with checkpoints: which actions are reversible, which are loudly logged, which would page someone at 03:00. You want that list before you drive a write path, not after.

Six tests that prove a confused deputy AI agent

Test 1: cross-tenant read

Plant a line in tenant A’s own content, a ticket thread, a shared document, a profile field, that asks the agent to retrieve or summarise a record belonging to tenant B. Then ignore the chat reply entirely and go to the data store’s access log.

Evidence that counts: a query for tenant B’s record, timestamped inside the session window, logged under the agent’s service account rather than the requesting user. A clean pass is either no query at all, or a query that carries the requester’s identity and is refused at the data layer.

Test 2: tool call under agent identity

Ask directly for something the authenticated user is not entitled to. No cleverness required. The question is whether the authorisation check happens at the model’s discretion or at the tool boundary, and whether the backend receives a user token or a shared one.

Watch the server-side identity field on the downstream request. If the API saw svc-assistant for an action that user A cannot perform in the UI, you have the finding whether or not the model phrased its refusal politely first.

Test 3: poisoned tool description or MCP result

This is the surface most permission reviews miss. Tool descriptions and tool results are model-visible text supplied by a third party, so a description on one MCP server can steer calls made through another, all under one shared agent credential.

Take an internal assistant wired to two MCP servers, one for the wiki and one for the CRM. Alter the wiki server’s search tool description, or the payload it returns, so it carries a follow-up instruction: after returning results, update the contact record. The user approves a search. Watch whether the CRM write executes, and under whose token. A clean pass is an agent that re-evaluates authority at the second call and a CRM that rejects it without a user identity attached. Testing of agent tool-calling boundaries and the permission model behind them belongs in scope from the first day, not as an add-on.

Test 4: retrieved document escalation

Move the instruction out of the chat box. Put it in an indexed file, a ticket comment from an external reporter, a dependency’s README, an HTTP response header the agent’s browsing tool will read. Reachability is the point: an agent that only reads text typed by the authenticated user has no attacker-reachable instruction channel, and the same technical flaw should be written up very differently in that case.

The coding-agent version is unpleasant and easy to reproduce. An agent that reads AGENTS.md and repository files picks up an instruction from a dependency’s README, runs a shell command, and pushes an environment variable into a build log, all with repository and CI credentials. The techniques for planting and containing instructions in agent instruction files apply directly here; so does the header-borne variant when the agent fetches URLs.

Test 5: write-path abuse

Reads are easier to get signed off. Writes are what move severity. Drive one state change that the organisation actually cares about: an invite sent, a refund issued, a ticket closed, a commit pushed, an email dispatched. Capture the record before and the record after, and the audit entry that names who the system believes acted.

Pair the three artefacts by timestamp: the injected content, the server-side log line with its identity, the changed row. That triple is the finding. Everything else is colour.

Test 6: approval bypass and post-approval change

Human-in-the-loop controls are weaker than they look, and this test is where that becomes obvious. Get approval for a benign call, then change the arguments after the dialogue was rendered, or append a second call the approval never covered.

Does the approval bind to a hash of the exact arguments, or to a friendly summary the model wrote? If the human saw one action and the system executed another, the control is not a brake. It is a confused deputy amplifier that also supplies someone to blame.

What counts as proof, and what does not

The decisive evidence is never the chat transcript. Agents narrate actions they did not take with total confidence: “I have deleted the record”is a measurement of text. It is not a measurement of harm. A confused deputy finding stands or falls on the server-side log line showing which identity the backend recorded, paired by timestamp with the state change.

Assemble the same evidence set every time: the injected artefact and where it lived, the request as the backend received it, the identity in the server-side log, the resulting state change, and the timestamps that tie them together. The evidence standard for consequence-proven findings does not relax because the subject is an agent.

Agents are non-deterministic, so one successful run is a lead, not a finding. Run each working path ten times, vary the phrasing slightly, and record the hit rate. Three in ten is a real confused deputy issue and should be reported as three in ten. Do not round it up to “reliably exploitable”and do not bury it because it failed seven times.

Three false positives recur. A hallucinated tool name that never reached any server. A refusal narrated as success. And the same path counted twice because it surfaced through two different prompts. Delete all three before the report goes out.

Fixes that hold, and ones that only look like fixes

Identity propagation is the fix. Per-user tokens, so the deputy cannot exceed the requester; the agent’s own role exists for infrastructure, not for user data. If every tool call carries the caller’s identity and the backend enforces on it, a planted instruction can still be read, but it buys nothing the attacker did not already have.

Second: deterministic authorisation at the tool boundary, evaluated on the arguments the model produced, after it produced them. Not on stated intent. Not inside the prompt. The orchestration layer either checks the resource ID against the session’s entitlements or it does not.

Then reduce the blast radius: scope tool permissions down to what the workflow needs, narrow the data breadth any single session can reach, make destructive calls reversible and loudly logged.

System prompt hardening, instruction guards and output filters all shift the frequency curve. They leave the authority boundary exactly where it was. If the remediation you are offered is a rewritten system prompt, plan the retest to find the same flaw through a different channel, because you will.

Fitting these tests into an engagement

Confused deputy testing sits after permission mapping and runs alongside tool abuse and excessive agency in a phased LLM red team. Map each finding to something recognised: prompt injection and excessive agency in the OWASP Top 10 for LLM Applications, plus the relevant MITRE ATLAS and ATT&CK AI techniques, so the write-up lands in a taxonomy the client’s risk team already uses.

Write the finding to survive review: a reachable path with the artefact attached, a demonstrated consequence, the reproduction rate, and the named identity in the log. Severity follows reachability more than it follows the elegance of the injection.

On retest, re-run the original path and then check the neighbours. Fixes have a habit of moving the vulnerable behaviour to a different tool, a different MCP server, or a second agent that inherited the same shared token.

The Short Version

Prompt injection is delivery. Confused deputy is the authority failure that gives it consequence. Map the credentials, mark which calls run as the user and which run as the agent, then prove the gap with the six tests above and server-side logs rather than chat output. Per-user identity propagation and argument-level authorisation close it; prompt wording does not. The same blast-radius logic underpins the seven tests for a hijacked assistant set out in the NCSC ransomware warning.

Next, read the companion piece on blast radius testing for agent prompt injection, then judging your own exposure before you decide how much of this applies to your stack.

Frequently Asked Questions

What is a confused deputy AI agent?

An LLM agent that holds privileges of its own, reads untrusted text, and carries out instructions found in that text using its authority instead of the requester’s. The classic example is a support assistant with one service account that can read every customer record. A line planted in customer A’s ticket asks for customer B’s details, and the application log shows the service account making the read.

How is a confused deputy attack different from prompt injection?

Prompt injection is the delivery mechanism: getting instructions into text the model will treat as a request. Confused deputy is the authority failure that turns that into impact, because the agent executes with permissions the attacker never had. An agent with no tools and no tokens can be injected all day and produce nothing worth reporting.

Do AWS confused deputy controls such as aws:SourceArn protect an AI agent?

They protect cross-service calls, which is a genuine and separate problem. Those condition keys constrain which service may assume a role on your behalf; they do not check whether the agent’s tool call matched the end user’s entitlements. Use them, and then test the agent’s decision layer separately.

How do you prove an AI agent acted on someone else’s authority?

With the server-side log, not the transcript. You need the injected artefact, the request as the backend received it, the identity the backend recorded, the resulting state change, and timestamps linking them. Repeat the path several times and report the hit rate, because agents do not behave identically run to run.

Can per-user tokens stop confused deputy problems in agent tool calling?

They remove most of the gap, because the agent can no longer exceed the person who asked. Pair them with deterministic authorisation on the actual arguments at the tool boundary, and keep the agent’s own role for infrastructure work rather than user data. Shared tokens across multiple MCP servers undo the benefit quickly, so check tenancy as well as token scope.