There is now a crowded market of tools that promise to bring AI to penetration testing. Some genuinely change how the work is done. Some are a thin wrapper around a language model with a security logo on top. Telling them apart matters, because the wrong tool does not just waste money, it produces findings you cannot trust. This is an attempt to sort the categories by how well they actually hold up in practice.
What Works
Code-reading assistants. Pointing a capable model at a codebase to find suspect patterns, explain unfamiliar logic and suggest where to look is a real gain. It does not replace review, but it directs attention well, and a human confirms each lead. This is the most mature and useful category today.
Output summarisers. Turning the raw firehose of scanner and tool output into a ranked, readable summary is dull, valuable work that models do well. The underlying data is real, measured by the scanner, and the model only organises it, which is exactly the safe division of labour.
Reporting aids. Drafting the write-up, translating a technical finding into language a client’s board will act on, and keeping tone consistent across a long report: this saves real hours at the least enjoyable stage of the job, with a human editing the result.
What Is Oversold
Autonomous “AI pentester” platforms. The promise is a system that tests a target end to end with no human. In reality these produce a mix of genuine findings, false positives stated with total confidence, and missed issues a competent tester would have caught. They can be a useful first sweep, but the output is a draft to verify, not a report to send. Sold as a replacement for a tester, they are a liability; sold as a force multiplier for one, they can help.
Anything claiming to “find all vulnerabilities with AI”. The confident, comprehensive-sounding pitch is the tell. Security is contextual, and a tool that does not understand what a system is for cannot judge what matters about breaking it. Breadth without judgement produces long reports and little insight.
The Question to Ask of Any Tool
One test cuts through most of the marketing: where do this tool’s facts come from? If the underlying findings are produced by real measurement, a scanner, a request, an actual interaction with the target, and the model only organises and explains them, the tool is probably sound. If the model itself is generating the findings, the tool will hallucinate, and its confidence will make the hallucinations hard to catch. Facts from measurement, analysis from the model. Any product that blurs that line should be treated with suspicion however polished the interface.
A Note on Trust and Data
There is a practical caution beyond accuracy. Feeding a client’s code, findings or infrastructure detail into a third-party AI service is a data-handling decision, and often a contractual one. Know where the data goes, whether it is retained, and whether it might be used for training, before it leaves your control. A tool that improves your speed while quietly exporting a client’s secrets is not a good trade.
The Short Version
The AI security tools that work are the ones that let the model organise and explain findings that real measurement produced: code-reading assistants, output summarisers and reporting aids. The ones that are oversold are the autonomous platforms that let the model generate the findings, because it will invent them with total confidence. Ask any tool where its facts come from, and mind where your client’s data ends up.