Using ChatGPT for penetration testing is not one activity, and confusing the two halves of it is the first mistake. One half is reading: condensing documentation, explaining an unfamiliar protocol, turning three days of scrappy notes into prose a client can follow. The other half is fact-finding: asking the model what is running on a host, which CVE applies, whether a header is missing. The first half is genuinely useful. The second half produces text that looks exactly like evidence and is not evidence at all.
Everything below follows from that split. Six limits, each with the failure mode, the tell, and the change to the workflow that contains it.
What ChatGPT actually does well in a penetration test
It reads well. Hand it a 90-page vendor integration guide and ask which authentication flows are supported, and it will sift the thing faster than you will. Ask it to explain an unfamiliar framework, a protocol handshake, a build system you have not touched in years, and you get a serviceable orientation. Regex, jq filters, small parsing scripts for odd log formats: all fine, because you run them and see the output.
Report drafting is where most testers get real value. Rough notes in, structured prose out, then you rewrite it against the evidence. That is reading and restructuring material you already hold, not producing new facts about a target.
There is a separate category that gets conflated with the chat window, and it should not be. Orchestration projects such as PentestGPT wrap a model in task tracking, tool execution and state, so the model steers while real tools do the measuring. Whatever you make of those projects, a chat tab is not one of them. A chat tab has no tools, no state and no contact with the target.
So the question to ask of any output before it leaves your hands is short: where did this fact come from, a model-generated assertion or a real measurement? Applied honestly, that question removes ChatGPT from the evidence path almost entirely and leaves it doing the reading. Which is what it is good at.
Limit 1: It produces assertions, not measurements
A model will tell you a port is open. It will tell you a version is vulnerable, that a security header is absent, that an endpoint accepts unauthenticated requests. It has touched nothing. The sentence is generated because sentences of that shape follow the prompt, not because anything was observed.
The damage shows up in drafts. A finding cites a CVE identifier with a plausible CVSS vector, scored against the FIRST CVSS specification, for a product version the tester never confirmed. The identifier has the right format. The vector string parses. The advisory does not exist for that version, or exists for a different component, or was withdrawn. Nobody catches it internally, and the client’s patching team catches it instead, which is a worse place to find out.
The rule is absolute. Every factual claim in a finding traces to a tool output, a request and response pair, or a screenshot you can reproduce on demand. If it traces to a conversation, it is not a finding yet.
This bites hardest in reconnaissance, because recon is where the temptation is strongest and the verification loop is loosest. There is a workable discipline for that, covered in more depth in this site’s guide to using large language models for reconnaissance, and it rests on the same principle: the model may organise and explain, but a tool that actually queried the world must supply the fact.
Limit 2: Non-determinism makes it a poor source of truth
Ask the same question twice and you may get two different answers. That is how sampling works, and no temperature setting in a consumer chat window makes it go away reliably. For ideation it is a feature. For evidence it is disqualifying.
What it quietly destroys is the negative finding. “I asked the model and it found nothing”is not coverage. Statements about what was not found are the most load-bearing sentences in a report, because clients act on them, and they require a repeatable method: the same scan, the same wordlist, the same authenticated session, producing the same result when a reviewer re-runs it next Tuesday.
If the model contributed anything to the thinking, log it. Prompt, response, date, and what you then verified with a tool. A reviewer should be able to see the join between the two. Keep the model out of any part of the methodology that has to be repeatable, and the non-determinism stops mattering.
Limit 3: Sending client data to an LLM is a contractual question first
Consider what actually leaves your laptop when you paste. Scope documents. Internal hostnames and IP ranges. Configuration snippets with credentials still in them. Architecture diagrams described in text. The client’s name, sitting in the same prompt as a list of their unpatched services.
Under UK GDPR and the Data Protection Act 2018, any personal data in that paste makes the service a processor in your chain, and the ICO’s guidance for organisations is the place to start on what that requires. But the contractual question usually arrives before the statutory one. Does your engagement letter or NDA permit third-party processing of client material at all? Many say, in plain terms, that engagement data stays within agreed systems. A consumer chat account is not an agreed system.
The account tier matters, and it matters less than people assume. Business and enterprise tiers typically carry different retention and training defaults from consumer accounts, and consumer accounts generally offer a training opt-out somewhere in the settings. Terms change. Read the current ones for the specific product and tier you are on rather than repeating what was true last year, and record what you read and when.
A sanitisation habit that holds up: tokenise hostnames and client names before pasting, strip credentials and tokens entirely, and keep raw tool output out of the chat window because raw tool output is where the identifying detail hides. If sanitising a paste takes longer than doing the work yourself, do the work yourself.
Limit 4: The chat window is itself an injection surface
Testers spend the week proving this against client systems, then do it to themselves at lunchtime.
Pasting a target’s page source, HTTP responses, JavaScript bundle, error output or repository README into a chat session puts attacker-controlled text into a system that cannot tell its instructions apart from the data it reads. That is the whole vulnerability. It is the same flaw this site covers in detail under prompt injection, and it does not care that the operator is a security professional. Filters and safety training blunt the obvious attempts. They do not remove the property, because the property is how the model reads.
A realistic version: you paste a minified bundle to get it summarised, and buried in a comment block is a paragraph of instruction-shaped text aimed at whatever model eventually reads the file. The summary comes back subtly steered, omitting a section or asserting something about the application’s auth flow that nothing in the code supports. You get no warning. The output looks like all the other outputs.
In a plain chat window the consequence is a polluted summary. In an agentic setup with browsing, code execution or connected tools, the consequence is a confused deputy acting with your permissions and your network position. Scope the permissions of any session you feed target data into on the assumption that it will be fooled, which is the practical reading of securing AI agents and their tool use. Treat every pasted artefact as untrusted. Keep the high-privilege session and the target-data session separate.
Limit 5: Refusals, guardrails and the exploit-writing gap
Refusal behaviour tracks keyword framing more than it tracks legitimacy. The same request phrased three ways gets a refusal, a lecture and a full answer. Chasing that around for an authorised engagement is billable time spent on prompt wording rather than on the target, and it produces nothing you could show a reviewer.
The worse failure is quieter. Ask for a working exploit against a specific service version and you can get code that compiles cleanly, imports the right libraries and reads like something from an advisory. It fails against the actual build, because the offsets were guessed, the API changed two releases ago, and nothing in the chain was ever validated against a running instance. Training cut-offs make this routine: new advisories, renamed tool flags, deprecated endpoints.
Exploitation belongs in your lab, against a real target, with real output in front of you. The model can explain the bug class and sketch the shape of the proof. It cannot tell you whether the thing works, and it will not tell you that it does not know.
Limit 6: It has no model of your client’s risk
Severity is a consequence judgement, not a text judgement. A model can write a persuasive high-severity paragraph about behaviour that changed nothing in the system, because the paragraph is assembled from the shape of severe findings rather than from what the application did. Grade findings on observed consequence: what data came back, what state changed, what a real attacker gains from the access that was actually demonstrated.
Remediation advice has the same problem. Generic hardening steps land badly against a real architecture, and worse against a real compliance obligation. Advice to rotate keys on a managed service the client does not control, or to enable a control their platform does not expose, costs you credibility in the readout meeting.
What a reviewer should be able to check in any AI-assisted write-up is simple enough to put on a checklist: every factual claim has a tool artefact behind it, every severity rating has an observed consequence behind it, and every remediation step is possible in that client’s environment. This site’s walkthrough of verifying AI generated pentest findings goes through those checks one at a time.
A workflow that keeps the model in its lane
The division is not complicated. The model reads and synthesises. Tools find facts. The tester verifies and signs off, and the judgement about severity, exploitability and what the client should do first stays firmly human, however much that dents the demo.
Three habits make it stick. Decide before the engagement what may be pasted, from which account, and write that decision into the test plan so it is a reviewable choice rather than a lunchtime one. Keep a prompt and response log for anything that fed the analysis. Treat drafted report text as a first draft that gets reconciled line by line against evidence, deleting any sentence that cannot be traced.
There is a great deal of noise in this space, and much of what is sold as AI-assisted testing is a chat window with a workflow diagram in front of it. If you are weighing up where tooling genuinely helps, this site’s assessment of what works and what is hype in AI-assisted pentest tools is the next thing worth reading.
The Short Version
ChatGPT is a reading and writing instrument for penetration testing, not a measuring instrument. It asserts; it does not observe. Its answers vary between runs, so it cannot support coverage claims or negative findings. Pasting engagement data into it is a contractual and UK GDPR question before it is a technical one, and pasting target-controlled content into it makes the session an injection surface aimed at you. It writes exploit code it has never tested, and it holds no model of your client’s risk. Used with discipline, it clears drudgery out of the way. Used carelessly, it puts fabricated facts into a document with your firm’s name on it.
Frequently Asked Questions
Is using ChatGPT for penetration testing allowed under most client engagement terms?
It depends entirely on the wording, and many engagement letters and NDAs restrict client material to agreed systems or named subprocessors, which a consumer chat account is not. Check the clause before the engagement rather than after a paste, and raise it with the client if the position is ambiguous. Approval for sanitised use with no client-identifying data is often achievable; retrospective approval is not.
Can ChatGPT find vulnerabilities on a live target?
No. A chat session has no contact with the target, so anything it says about a live system is a model-generated assertion rather than a measurement. Tooling that does reach the target, including orchestration projects that drive scanners and run commands, is a different product category and should be judged on its own terms.
What should never be pasted into a ChatGPT session during an engagement?
Credentials, API keys and tokens, raw configuration files, client names paired with vulnerability detail, internal hostnames and address ranges, and anything containing personal data. Raw tool output is the common leak, because it carries identifying detail the tester has stopped noticing. If a paste needs more than a quick redaction pass, it probably should not be pasted.
Can pasting target output into ChatGPT cause a prompt injection problem?
Yes, and it is the limit people overlook most. Page source, HTTP responses, JavaScript and repository files are attacker-controlled text, and the model cannot separate instructions from the data it reads. In a plain chat the result is a steered or polluted summary; in a session with browsing, code execution or connected tools, the model acts with your permissions.
Does ChatGPT write usable exploit code for a penetration test?
It writes code that often compiles and sometimes reflects the right approach, but it has validated nothing against the target version. Offsets, library behaviour and API surfaces change, and training cut-offs mean recent advisories and flag changes may be missing. Treat any generated proof of concept as a sketch to be built and proven in your own lab.
