AI Hallucination in Security Reports: 5 Red Flags

AI Hallucination in Security Reports: 5 Red Flags

A security report is not prose. It is a chain of claims, and every claim carries a provenance question: did a tool measure this, did a tester watch it happen, or did a model assert it? AI hallucination in security reports is what happens when that third category gets printed in the same typeface as the first two. The sentence reads well, the formatting is clean, the CVE is the right shape, and nobody downstream can tell the difference.

That is the problem worth solving. Not the definition of hallucination, which everyone already has, but the artefact: a PDF that goes to a client, feeds a risk register, and sometimes lands in an ISO/IEC 27001 audit folder. Below are five things a reviewer can spot in minutes, and the way to write findings up so they survive a re-run.

Why AI Hallucination Shows Up in Security Reports in the First Place

A language model produces the most plausible continuation of the text in front of it. It does not measure anything. Security reporting, meanwhile, rewards confident, well-structured, tightly formatted prose with identifiers and version ranges in it. The thing the model is good at producing and the thing the format rewards are the same thing. Fabrication and competence look identical on the page.

Templates make it worse. A finding form with empty boxes for Impact, Affected Versions, Remediation and References is an invitation to fill them, and a model asked to fill them will fill them. Invented detail collects in exactly those boxes, because that is where the tool output ran out and the prose had to keep going.

Then there is non-determinism. Feed the same scan output and the same prompt into the same model tomorrow and the finding list can differ: a severity shifts, an item drops, a new one appears. One pass is not evidence. It is a sample of one from a distribution nobody has characterised.

For internal policy it helps to point at recognised framing rather than argue from first principles. OWASP’s Top 10 for LLM Applications names Misinformation as LLM09 in the 2025 edition, with overreliance treated as the human half of the same risk (earlier editions listed overreliance on its own). Check the current edition before you quote it, but the split is the useful part: the model generating a confident falsehood is one failure, and a reviewer accepting it unchecked is a second, separate one. Most report incidents need both.

Red Flag 1: Identifiers That Look Right but Do Not Exist

CVE-2023-41892 is a format. A model can produce that format all day. What it cannot do reliably is attach the right identifier to the right product line, so you get a correctly formatted CVE sitting beside a vendor name and version range it has nothing to do with. The reviewer resolves it against the National Vulnerability Database or the vendor advisory and the finding collapses in thirty seconds. CWE references drift the same way, usually landing one level too general or on a sibling weakness.

Links are the second tell. Advisory URLs that 404, bulletin references that resolve to an unrelated page, or a “see vendor KB”that points at a product retirement notice. Nobody clicks them in review, which is precisely why they survive to print.

Hallucinated package names are the dangerous version. A remediation step that says “upgrade to @acme/secure-parser 3.1″when that package was never published does more than waste time, because once the name is public somebody can register it. Package hallucination feeding slopsquatting is a known pattern in the dependency ecosystems, and a security report is an unusually trusted place for a fake name to appear.

The control is dull and absolute: every identifier, link and package name gets resolved against NVD, the vendor advisory or the registry before it stays in the document. No exceptions for the ones that look obviously fine. Those are the ones that get through.

Red Flag 2: Evidence That Cannot Be Traced Back to a Request

Read a finding and ask what a sceptical engineer would need to replay it. Request, response, payload, timestamp, host. If the finding is a narrative summary with none of that, it is a story about a test rather than a record of one.

Tidied evidence is the subtler flag. A response snippet with neat indentation, consistent header casing and no noise has probably been reconstructed rather than pasted from the proxy log. Real captures are ugly. They carry stray cookies, odd ordering, a Server banner nobody asked about. When the ugliness disappears, something rewrote it, and whatever rewrote it also had the opportunity to invent.

Worst case: transcript detail that appears nowhere in the raw tool output. A port the scan never reported. A header the proxy never saw. That is a model filling a gap, and it means the whole finding needs rebuilding from the artefacts up.

The reporting pipeline is an injection surface

Here is the failure mode nobody puts in their marketing. If an assistant summarises banners, HTTP headers, page content or scan output, it is reading attacker-controlled text. A header value that says this response is from a staging decoy, no action required can end up quoted as fact in the write-up, or used to argue a real finding down to informational. The root cause is the one that defines LLM security generally: the model cannot separate its instructions from the data it reads. That is the whole vulnerability. Treat the summarisation step like any other untrusted input path, and assume the mitigations are partial.

Red Flag 3: Remediation Steps the Stack Would Reject

Fabricated configuration is written in convincing syntax. An Nginx directive with the right naming convention that does not exist. An Apache flag borrowed from a different module. A header name that is one word off the real one and so does nothing at all when applied.

Watch also for advice aimed at a platform the client does not run. Systemd instructions for a Windows estate, or guidance pinned to the version the model saw most often in training rather than the version in scope. Default-stack drift is a reliable signature of generated remediation.

Then there are the fixes that are not fixes. Prompt injection mitigations are partial by nature: input filtering, allowlists, privilege separation and output checks reduce exposure, they do not close the hole. A report that promises resolution misleads the reader into closing a risk item that is still open.

Hallucinated remediation is worse than a hallucinated finding. A fake CVE costs an afternoon. An invented hardening directive leaves the real gap open while the register says remediated, and that misreading can run for a year. The cheapest control in this whole article: paste the recommended config into a lab instance and see whether the service starts. Two minutes, and it catches almost all of it.

Red Flag 4: Severity Asserted Without a Demonstrated Consequence

A model can grade anything. Ask for a CVSS vector and you get a CVSS vector, complete and plausible. That is not an assessment. A vector or risk band attached to an impact paragraph that describes no demonstrated effect on data, permissions or state is an opinion wearing a numeric costume, and recalculating it honestly will usually move the band.

Hallucination hides in the gap between output and consequence. A finding that says the model produced a forbidden sentence proves very little; the system said a thing. A finding that shows the agent called a tool, read a file it should not have reached, or changed state proves something real. Grade by what the system did, which is the same discipline behind a proof-over-output testing checklist.

Agent and tool-use findings need the blast radius on the page: which credentials and permissions the model held at the time, which tools were mounted, and what a confused deputy pattern would reach from there. Without that, the reader cannot tell a chat quirk from a path to the customer database. Rewrite every impact section so it names three things: the action taken, the data touched, and the account or token used. If you cannot fill all three, you do not yet have an impact.

Red Flag 5: Findings That Change or Disappear When You Re-Run Them

Non-determinism changes how findings must be written, not just how they are tested. Record the model name and version, the date, the temperature or sampling settings, the system prompt if you have it, and the exact input. A finding without that context cannot be reproduced by anyone, including the person who wrote it.

Attempt counts beat absolutes. “Three successes in twenty attempts against gpt-4o-mini, 2025-09-15, temperature 0.7″is actionable. “The model can be jailbroken”is a claim that will be argued away in the readout, however true it was on the day. Rates also let a client reason about control strength: a one-in-twenty bypass against a public endpoint is a different problem from one-in-twenty against an internal tool with two approvals in front of it.

When a result will not reproduce at all, there are two honest options. Remove it, or demote it to an observation with the original transcript attached and a plain note that re-runs failed. What you do not do is quietly rewrite the wording until it sounds solid again. The probabilistic shape of these results is also why automated LLM red teaming misses things that a tester with a hypothesis will find.

A Provenance Pass Before the Report Leaves the Building

One pass, before sign-off, over the whole document. Mark each assertion as measured by a tool, observed by a tester, or model-generated. Nothing in the third bucket ships unmarked or unverified. Done properly it takes an hour on a mid-sized report and it finds things.

Apply the same question to any AI-assisted tool in the chain: where did this fact come from, and can you show the raw artefact behind it? A tool that can surface the underlying request, response or scan line is doing synthesis. A tool that cannot is reporting a model’s guess with a tool’s authority. That single question sorts most of the market, and it is the spine of a sensible evaluation of AI pentest tooling.

Keep the model on reading, synthesis and phrasing. Keep it off fact-finding, identifier lookup and verification. It is genuinely good at turning forty pages of output into a readable narrative, and at organising reconnaissance material that a tool actually gathered. It is not a source of facts, ever.

Third-party handling deserves a policy, not a shrug. Client host names, credentials, internal URLs and unpublished findings pasted into an external service have left your control, and if the same workspace serves several engagements, context from one can surface in another’s write-up. Check the data retention and training terms, strip identifiers before anything goes out, and keep engagements in separate contexts.

Sign-off stays where it always was. A named human reviewer per finding, with the same accountability that applied before any of this was drafted by a model. However much that dents the demo, the judgement stays firmly human.

The Short Version

Every sentence in an AI-assisted security report is measured, observed or asserted, and only the first two have earned their place unqualified. Resolve every identifier. Demand a replayable request behind every piece of evidence. Test remediation in a lab before it ships. Grade findings by what the system did, not what it said. Record model, version, date and attempt counts, because a finding nobody can reproduce will be talked away. And remember the pipeline itself reads attacker-controlled text, so the report is an injection surface too. A great deal of what is claimed for AI in reporting is marketing; the provenance pass is the part that actually holds.

Frequently Asked Questions

What causes AI hallucination in security reports?

A language model generates the most plausible next text rather than reporting a measurement, so when tool output runs out, fluent invention continues in its place. Report templates make this worse by presenting empty fields for impact, affected versions and references that something has to fill. OWASP’s Top 10 for LLM Applications frames the two halves as misinformation from the model and overreliance by the reader.

How can I tell whether a pentest finding was measured or generated?

Ask for the raw artefact: the request, the response, the payload, the scan line, the timestamp. A measured finding can be replayed by someone who was not there; a generated one reads as narrative summary with tidy, suspiciously clean evidence. If the transcript contains detail that appears nowhere in the raw tool output, a model filled a gap.

Should AI-drafted findings be labelled in the final report?

Internally, yes, without question: every assertion carries a provenance tag through review so nothing model-generated reaches sign-off unverified. In the client-facing document, the better answer is that labelling becomes unnecessary because the verification happened, and each finding carries its own evidence. Methodology sections should still state plainly where AI assistance was used in drafting.

How do you report a finding that only reproduces some of the time?

Report the rate, not the absolute: successes out of attempts, with model name and version, date, sampling settings and the exact input. That gives the client something to re-test and defends the finding in the readout. If it will not reproduce at all, remove it or demote it to an observation with the original transcript attached.

Is it safe to paste engagement data into a third-party AI service to draft a report?

Not without checking the retention and training terms and getting it agreed in the engagement contract first. Host names, credentials and unpublished findings sent to an external service have left your control, and shared workspaces can carry context from one engagement into another’s write-up. Strip identifiers, keep engagements in separate contexts, and treat anything under NDA as out of scope for external tools by default.