Ask what is an AI agent and the answer you get back is almost always a list of capabilities: it perceives, it reasons, it decides, it acts. Accurate enough, and useless for deciding whether to sign the thing off. Once an agent is live, its capabilities matter far less than its permissions. A model that can reason brilliantly and call nothing is a curiosity. A mediocre model holding a refund API and a service token is a financial control.
So the definition worth keeping is narrower and more awkward: an AI agent is a language model that has been given a goal, a set of tools it can call, and permission to act without a human approving each step. Everything that follows comes from that last clause.
What an AI Agent Is, in Plain Terms
A working definition
Three ingredients make an agent. A goal, expressed in natural language rather than code. Tools, which are functions the model can invoke to read or change something in the world. And standing permission to use those tools, so the loop turns without a person clicking approve between each turn.
Remove the tools and it is a chatbot. Remove the standing permission and it is a very chatty approval queue. Keep all three and you have software whose control flow is decided at runtime by a statistical model reading text.
Four things that separate an agent from a chatbot
- A goal rather than a turn. A chatbot answers what was just said. An agent is handed an objective (“resolve this ticket”, “fix this failing test”) and works towards it across many steps.
- A loop rather than a reply. The model’s output feeds back into its own input, along with whatever its tools returned, and it decides again.
- Tools rather than text. The output is not only prose. It includes structured calls to functions that query databases, write files, send email or move money.
- State that persists. Scratchpads, conversation history, vector stores and task memory carry information between steps, which means content read at step two can influence a decision at step nine.
What an agent is not
A scripted workflow with a model in the middle of it is not an agent. If the order of operations is fixed in code and the model only classifies, summarises or drafts, the code path is still reviewable and the blast radius is still bounded by the script. That is a normal application with an unreliable function in it, and it should be tested as one.
A retrieval chatbot that answers questions over your documents is not an agent either, however much the vendor deck calls it one. It reads; it does not write. A single API call behind a nicer interface is not an agent. Confusing these with real agents is the first mistake, because it leads teams to apply content filtering to a system whose actual problem is a credential.
Autonomy is a spectrum, not a badge
Place any system on a line. At one end, the model suggests and a human executes. Next, the model executes reversible actions alone and queues irreversible ones. Next, the model executes everything within one system, under one scope. At the far end, the model plans multi-step work across several systems, spawns subtasks and decides for itself when it is finished.
Where a system sits on that line is a product decision, and it is usually made by accident. Someone adds write access to a tool because read-only made the demo look weak.
The Five Moving Parts Inside an Agent
The model and its reasoning loop
The model chooses the next step. Not a router, not a state machine, the model. It emits either a final answer or a tool call with arguments, the result comes back as text, and it chooses again. There is no fixed path through the system to review, which is the first thing that breaks conventional code review.
The instruction layer
System prompt, role description, policies, refusal rules, output format. All of it is text in the same window as everything else the model reads. It carries no special privilege at the architecture level, only a convention that the model has been trained to respect. Treat it as a strong default, not a boundary.
Tools and function calling
Tools are where capability becomes real. A tool is a function signature the model can fill in: get_order(order_id), issue_refund(order_id, amount), write_file(path, contents). Model Context Protocol servers have made this cheap to assemble, exposing repositories, ticket systems, file shares and internal APIs to an agent through one standard interface. Cheap to assemble also means cheap to over-scope, which is why testing the MCP layer itself matters as much as testing the model.
Memory and retrieval
Retrieval quietly converts outside content into input. A supplier’s PDF, a Confluence page, a web result, a ticket from last Tuesday: once retrieved, all of it sits in the context window as plain text, indistinguishable in form from the operator’s own instructions. Most teams think of retrieval as a quality feature. It is an input channel.
Identity and permissions
The last part is the one that decides everything. When the agent calls issue_refund, whose credential executes it? In most deployments the answer is a single service account provisioned during the pilot, holding more access than any individual user of the system. That credential, not the model’s cleverness, is the agent’s true capability.
How the Loop Runs: A Worked Example
Take a customer support agent. One cycle looks like this:
- It reads the ticket. Untrusted input, controlled by the customer.
- It plans: identify the order, check eligibility, decide on a refund.
- It calls get_order_history(customer_id). A read.
- It reads the result, which may itself contain customer-supplied text such as a delivery note.
- It decides.
- It calls issue_refund(order_id, amount). A state change. Money moves.
Only one of those six steps can cost anything. Mark the reads and the writes separately on your own diagram and the risk picture resolves immediately, because the reads are where attacker-controlled text enters and the writes are where consequence lives.
Now run the same ticket again. You may get a different sequence: an extra lookup, a clarifying question, a refund at a different amount, or no refund at all. The model samples. That non-determinism is not a defect to be ironed out; it is a property of the system, and it means there is no single code path to review and no single run that proves anything.
Worth noting what the evidence actually is. The chat transcript shows what the agent said. The tool call logs show what the agent did. Only the second one proves what happened, and in a well-built system the two will sometimes disagree, because a model can describe a refund it never issued and issue one it never mentioned.
Why AI Agents Break Differently From Normal Software
Instructions and data arrive through the same channel
A language model reads one stream of tokens. The system prompt, the user’s message, the retrieved document and the tool output all arrive as text in the same window, and the model has no reliable mechanism for deciding which of them has authority. Any content it reads is so a candidate instruction. That is the whole vulnerability, and prompt injection follows directly from it.
A recruitment screening agent parsing CVs shows it cleanly. White text in a PDF, or a line buried in document metadata, reaches the model exactly as the candidate’s genuine experience does. The model does not see formatting. It sees tokens.
Guardrails, classifiers and hardened system prompts are worth having. They narrow the window. They do not close it, because the confusion is in how the model reads, not in what it was told, and anyone selling a product that claims to eliminate prompt injection is selling confidence rather than a fix.
The confused deputy pattern
An agent typically holds broader access than the person talking to it. A support agent can refund any order; the customer can only submit a ticket. When injected text succeeds, the resulting action inherits the agent’s permissions, not the requester’s. The deputy is confused: it acts with its own authority on someone else’s instruction. Proving the deputy can be fooled, rather than assuming it, is the point of turning AI techniques into working tests, as set out elsewhere on this site.
The scoping question teams skip is whether tool calls carry the end user’s identity or the agent’s shared service account. If it is the shared account, every user of the system effectively has the agent’s full access, mediated only by the model’s judgement. This is where the serious findings usually are.
Excessive agency
Write access granted alongside read because it was convenient. A delete endpoint exposed because the API client had it anyway. Email send enabled for notifications and reachable for anything. The OWASP Top 10 for LLM Applications treats prompt injection and excessive agency as separate entries for good reason: the first is how a bad decision gets made, the second is how far it reaches.
Mapping the Attack Surface Before You Trust an Agent
Before any testing, write down four things: what the agent can read, what it can change, whose credential it uses, and where its outputs land. Most teams can answer the first two and stall on the third.
Then classify every input channel by who controls it. The end user. A colleague in another department. A supplier’s invoice PDF. A GitHub issue comment from an outside contributor. A public web page the agent retrieves mid-task. The further down that list you go, the less the content has been through any review, and the more it should be treated as hostile by default.
Next, list the consequential actions in plain language: payments issued, records deleted, email sent outside the organisation, files written, code merged, tickets closed on a customer’s behalf. Those are your findings-in-waiting. Our guide to scoping AI penetration testing properly works through how to turn that list into a test plan.
Finally, the part that gets deferred to a later compliance pass and should not be. Handing an agent live data means handing data to a hosted model provider. Retention periods, processing region, subprocessors and whether the provider trains on inputs belong in the design review next to the permissions model, because both decisions are being made at the same time and only one of them is easy to change later.
Designing and Checking an Agent on the Assumption the Model Will Be Fooled
Design so that a fooled model is survivable. Least privilege per tool rather than per application. Per-user identity on tool calls, so the agent cannot exceed the authority of whoever it is acting for. Human approval on anything irreversible, which in practice means money, deletion and outbound communication. The NCSC and CISA Guidelines for Secure AI System Development, published in November 2023 with international co-signatories, cover the design-stage version of this well and are worth reading before architecture is settled rather than after.
Separate reading from acting. Content pulled in from outside should not be able to trigger a state change in the same loop iteration without passing through a check the model does not control. Deterministic code, an allowlist, a second credential, a human. Something that is not the model.
When you test, judge by consequence rather than output. A model producing a forbidden sentence is noise. A refund issued, a record overwritten, an email delivered outside the organisation, a file moved out of the trust boundary: those are findings, and they live in the tool call logs. For the coding agent case, picture an MCP server with repository read, write and issue access, where a comment on an open issue becomes an instruction and the agent’s own token performs the commit. The vulnerability is proved by the commit, not by the agent agreeing to do it.
Non-determinism changes what counts as evidence, in both directions. One successful exploit is a finding. One clean run is not a pass. Repeat the run, record the tool call sequence and the resulting state change, and report how often the path was reachable. A single clean attempt is weak evidence and should be written up so, however much that dents the demo. Our notes on securing agents and their tool use go further into the controls; the wider explainers library covers the underlying concepts in more depth.
The Short Version
An AI agent is a language model with a goal, tools and standing permission to use them. Its risk is not set by how clever it is but by what it can do while it is wrong. Instructions and data reach the model through one channel, so anything it reads can act as an instruction, and filters narrow that window without closing it. Most deployments compound the problem with a single service account that outranks every human user. Scope the permissions, separate reading from acting, prove findings with tool call logs and repeat runs, and the honest answer to what is an AI agent becomes a question about credentials rather than capability.
Frequently Asked Questions
What is the difference between an AI agent and a chatbot?
A chatbot produces text in response to a turn of conversation. An agent is given a goal, a set of tools it can call and permission to act across multiple steps without approval at each one. The practical difference is that a chatbot can be wrong, while an agent can be wrong and then do something about it.
Does an AI agent need to be fully autonomous to count as an agent?
No. Autonomy is a spectrum, from suggesting actions a human executes, through executing reversible actions alone, to planning multi-step work across several systems. Anything holding a tool and a credential it can use without per-step approval should be treated as an agent for security purposes, whatever the product team calls it.
What is the biggest security risk with AI agents?
The combination of prompt injection and over-broad permissions. Injection is the mechanism by which content the agent reads becomes an instruction; the permissions decide how far the resulting action reaches. The confused deputy case, where the agent’s shared service account holds more access than the person talking to it, is the pattern that turns a text trick into a real loss.
How do you test an AI agent before putting it in production?
Map permissions, input channels and consequential actions first, then attempt injection through each channel a third party can influence and check whether state actually changed. Evidence comes from the tool call logs rather than the chat transcript. Repeat every attempt, because one success is a finding and one clean run proves very little.
Can an AI agent be given access to company systems safely?
It can be given scoped access with acceptable risk, which is not the same as safely. Per-tool least privilege, per-user identity on tool calls, human approval on irreversible actions and a hard separation between reading outside content and changing state will keep a fooled model survivable. Nothing available today removes the underlying confusion between instructions and data.