Verifying AI Generated Pentest Findings: 6 Checks

Verifying AI Generated Pentest Findings: 6 Checks

Verifying AI generated pentest findings is now a standing job on most security teams, because the output arrives faster than anyone can read it. A platform, an agentic harness or a vendor’s AI-assisted engagement hands over a list of issues with severities already attached, remediation text already written and a tone of total confidence throughout. None of that confidence is evidence. Treat each line as a claim, and the question that sorts the real from the invented is always the same: did a tool measure this, or did a model say it?

The six checks below are the triage. They are ordered so the cheap ones come first, which matters when a run produces forty entries and you have an afternoon.

Why AI pentest findings land on your desk unverified

There are two failure modes, and confusing them is the first mistake. The first is fabricated detail: a plausible endpoint, a parameter that does not exist, a CVE number attached to a version that was never seen in a banner. The second is real but inconsequential output, where the model genuinely produced something it should not have said, and nothing in the system acted on it. The first is hallucination in security reports. The second is a measurement of the wrong thing.

Volume makes both worse. One authorisation flaw in a shared middleware check can emit as a dozen findings across a dozen API paths, each with its own severity, its own remediation paragraph and its own confident summary. Review capacity does not scale with generation capacity, so the backlog either gets rubber-stamped or gets ignored, and both outcomes are bad in different ways.

Then there is non-determinism. A tool that saw a behaviour once will often report it as a permanent property of the system. Temperature, session state, retrieval contents and the provider’s current model build all move the result. A finding with no reproduction record is a snapshot presented as a fact.

Your triage workflow is attack surface too

Pasting attacker-controlled page content, HTTP headers or agent logs into a model to summarise a finding puts hostile instructions inside your reporting pipeline. The payload you are investigating is text, and the summariser reads text the same way the target did. Injected instructions can steer the summary, bury a finding, or add one. On top of that, the paste sends engagement evidence to a third party, which is a data-handling decision under UK GDPR and the Data Protection Act 2018, and it needs making before the first paste rather than after. The mechanics of how untrusted text reaches a model through infrastructure are covered in more detail in this guide to indirect prompt injection in HTTP headers.

Check 1: trace every fact to a measurement, not an assertion

Apply the tooling test to individual findings, not just to the product. A platform that states “this endpoint is vulnerable to broken object level authorisation”is making a model-generated assertion until it hands over the stored request and response showing another user’s record in the body. A sentence describing an artefact is not an artefact.

Ask for the thing itself: the request and response pair, the tool-call log, the HTTP status, the database row, the file that was written, the timestamp on the audit entry. Then classify each line of the finding as observed, inferred or asserted, and mark every asserted line unproven until the artefact turns up.

The tells of a model-generated assertion are consistent. Version numbers and CVE references with no banner or response behind them. Remediation advice that names a framework the application does not use. A severity stated with total confidence and an evidence block that is one paragraph of prose. Where AI genuinely earns its place at this stage is the opposite task: reading and synthesising large volumes of evidence that a human has already collected, which is also the honest use case set out in this breakdown of using large language models for reconnaissance. Synthesis is fine. Fact-finding stays with the tester.

Check 2: reproduce it, then record how often it reproduced

Non-determinism changes what “fixed”means, so binary pass or fail is the wrong unit. Run the attempt a fixed number of times and report a rate. A reproduction record reads like this: 4 successes in 20 attempts, against model version X, system prompt hash Y, temperature 0.7, tool set Z, fresh session each attempt, tested 15 September 2026. That is a finding. “Reproduced successfully”is not.

Pin and log the variables that move the result:

  • Model name and build or version string
  • System prompt version or hash
  • Temperature and any sampling settings
  • The tool set exposed to the agent
  • Session or conversation state, and whether each attempt started clean
  • Retrieval corpus contents at the time of test
  • Date of test

A single clean retest does not close an AI finding. If the original rate was 4 in 20, one failed reproduction is the expected outcome most of the time, and reading it as a fix is arithmetic error dressed as good news. Worse, a provider model update silently invalidates both the finding and the retest that closed it. A rate measured at 4 in 20 before an update and 11 in 20 after it is the same vulnerability with a different likelihood, and the retest that passed in between proved nothing durable.

When something cannot be reproduced at all, keep it as an observed-once entry with the full transcript attached. Do not quietly delete it, because the next tester will rediscover it. Do not inflate it either.

Check 3: separate what the model said from what the system did

Testing to consequence sorts most inflated AI findings in minutes. Judge the finding by what changed: an email sent, a record read, a refund issued, a file written, a repository commit. A model producing a forbidden sentence is not a system acting.

Take a retrieval-backed support agent where the tool reports “successful data exfiltration”. Read the transcript and the model has recited a plausible-looking customer record: right field names, right format, convincing enough to alarm a manager. Read the tool-call log and there is no database read at all. The model produced the shape of a record from training data and context, and nothing was retrieved. That finding drops from high to informational, and the write-up should say exactly why.

Refusal-bypass findings with no tool behind them are usually content issues rather than security findings, and they should be scored that way. Trace the chain from injected instruction to tool call to side effect, and when the chain stops, show the reader where it stopped. The same jailbreak text is low value against a read-only chatbot and serious against an agent with write access to a ticketing system or a code repository, which is why the tool-use surface around an agent decides the rating more often than the prompt does.

Check 4: test the blast radius against the real permission map

Before accepting a severity, map what the agent can actually reach. Which identities does it act as. Which tools can it call. Which data sources does it read. What scopes sit on each token, and are they the scopes in the design document or the scopes in production.

This is the confused deputy pattern in its modern form: the attacker supplies the instructions and the agent supplies the credentials. Impact is bounded by the agent’s permissions, not the attacker’s, which means permission scoping is the control that changes the number. Prove reach with one concrete action at the edge of scope, a single write to a low-value record or a read of one row that should be out of bounds, rather than listing theoretical consequences in a paragraph of conditionals.

State the root cause plainly in the write-up: a model cannot reliably separate its instructions from the data it reads, so filters and guardrails reduce likelihood rather than remove the issue. That framing stops an engineering team closing the ticket with a prompt edit and leaves scoping on the table. Judging your exposure to prompt injection starts from the same premise. Map the finding to OWASP’s Top 10 for LLM Applications (LLM01 prompt injection, LLM02 sensitive information disclosure) and to MITRE ATLAS for technique mapping, then score with CVSS v4.0, published by FIRST in 2023, so the severity sits in a frame an auditor recognises.

Check 5: judge whether an engineer can act on the evidence as written

A finding that cannot be handed to a developer without a conversation is half a finding. The minimum pack per entry: the exact input, the full transcript or request and response pair, the tool-call log, a timestamp, the environment and model version, the reproduction rate, and the observed consequence.

Rewrite model-authored remediation that names the wrong control. The pattern repeats: content filters recommended where the fix is scope reduction, a prompt tweak recommended where the fix is a human approval step before the agent writes anything. Model-generated remediation tends to describe the general literature on a vulnerability class rather than this system’s actual control points.

State the limits of what was verified inside the finding itself. One line is enough: tested against staging, read paths only, not retested after the 20 September model update. A reviewer, a client or an auditor can then see the boundary instead of guessing at it. Handle evidence containing live payloads with care, including redaction, where it is stored, and whether any of it goes near a third-party service.

Check 6: collapse duplicates and downgrade what will not hold up

Group by root cause and shared control, not by endpoint or payload. Twelve AI-generated findings across twelve API paths become one authorisation flaw in a shared middleware check, with the affected endpoints listed as scope. The remediation is one change. Reporting it twelve times inflates the count, splits the fix across twelve tickets and makes the engagement look productive while making the problem harder to solve.

Every entry then gets one of four outcomes: confirmed, confirmed at lower severity, observed-once, or withdrawn. Keep a withdrawn log with the reason recorded, so the same fabricated finding does not reappear in the next run and so a client can see the filtering was deliberate rather than convenient. Set the rule and hold it: nothing reaches a ticket, a board paper or a client report without a named artefact and a reproduction record. The same discipline runs through this LLM testing checklist built around proof rather than output, and it is the single thing that separates a usable AI-assisted engagement from an expensive text generator.

The Short Version

Six checks: source, reproduction count, consequence, blast radius, actionability, deduplication. Ask where each fact came from and demand the artefact. Report rates, not pass or fail, and log the model version alongside them. Follow the chain to a side effect before you score anything high. Bound impact by the agent’s permissions. Rewrite remediation the model got wrong. Collapse duplicates by root cause, and withdraw what cannot be proved, in writing. The output volume is new; verifying AI generated pentest findings still comes down to evidence, and the judgement stays firmly human, however much that dents the demo.

Frequently Asked Questions

How do you tell an AI hallucination from a real pentest finding?

Ask for the artefact. A real finding comes with a stored request and response, a tool-call log, a database row or a written file, with timestamps that line up. Hallucination in security reports usually shows as confident prose with no evidence block, version or CVE numbers that no response confirms, or remediation advice aimed at a stack the application does not run.

How many times should a finding be reproduced before it counts as confirmed?

There is no fixed number that works everywhere, but use a fixed denominator and report the rate, for example 20 attempts with the successes counted. One success in 20 is still a real finding with a low likelihood; 20 in 20 is a different conversation. Record the model version, system prompt version, temperature, tool set and session state with the rate, because a provider update can change it without anyone touching the application.

Does a jailbreak or refusal bypass count as a security finding on its own?

Usually not. If the model produced forbidden text and no tool ran, no data moved and no record changed, it is a content and brand issue rather than a security vulnerability. The same bypass becomes serious the moment it reaches an agent that can write to a ticketing system, a repository or a payment flow.

Is it safe to paste client evidence into an AI service while verifying findings?

Not by default. Engagement evidence often contains personal data and attacker-controlled text, so sending it to a third-party service is both a processing decision under UK GDPR and the Data Protection Act 2018 and a route for injected instructions into your reporting workflow. Decide the data-handling position, retention terms and redaction rules before the first paste, and prefer self-hosted or contractually covered services for live client material.

Who is accountable when an AI-assisted report contains a fabricated finding?

The person or firm who signed the report. Tooling choices do not transfer accountability, and no vendor claim about validation survives a client asking where a particular finding came from. That is the practical reason for the standing rule on artefacts and reproduction records, and the reason task-by-task review of what AI can and cannot take over keeps landing on verification as the human part.