Why One AI Pentest Run Isn't Enough

Today's powerful AI and large language model services are non-deterministic. An identical input into the same AI system will yield different results that diverge over execution time. The more vague the input, the longer it needs to reason, and the more advanced the expected output, the greater the variance in responses. In cybersecurity terms, non-determinism translates to missed vulnerability findings, skipped methodology steps, and a fundamental lack of trust in both the quality of findings and the security integrity of the target. If you run one autonomous pentest and trust the results, you're making a bet you can't quantify. The security industry's well-established history of over-claiming AI precision — something I examined at length in The Cusp of a Virtual Analyst Revolution — is a useful reminder that "AI says so" has never been a substitute for validated methodology.

Let's be honest about something, though: humans aren't exactly sticklers for deterministic execution either.

The Non-Deterministic Human

Depending on what the consultant had for breakfast and how much sleep they got the night before, even the most qualified human brings a degree of randomness and non-repeatability to an engagement. This has never been a secret in the industry. Modern pentesting has developed a set of countermeasures to manage it:

• Teams of pentesters working together — splitting tasks, collaborating, confirming adherence to stated methodologies via checklists, and rotating between engagements to ensure a mix of skills and experience is applied over time.
• Extensive use of automated tooling to rigorously grind through standardized testing methodologies, output auditable logs, and enable manual confirmation of findings.
• Real-time collaboration and interaction with the customer to confirm and refine testing objectives and resolve blockages.
• Senior consultants directing, reviewing, and evaluating each pentester's performance and efficacy.
• Frequent training, co-development and refinement of new testing methodologies, tool selections, threat intelligence usage, checklists, and reporting templates.
• Independent review or editing of pentest reports by a senior consultant or program manager.

The cumulative effect of these practices is not determinism — it's sufficient reliability and auditability for customers to trust the results. Autonomous pentesting needs equivalent mechanisms.

How Autonomous Pentesting Addresses Non-Determinism

The evolving field of autonomous pentesting is developing a growing set of analogous techniques to increase execution reliability and customer trust. Examples being applied across the industry today include:

• Multi-agent architectures where pentest tasks and methodologies are split up, quantized, or divided amongst multiple narrow-focused agents to minimize hallucinations and prevent context deterioration over long engagements.
• Supervisor and governor agents that watch other agents and nudge them toward optimal performance and outcomes.
• Use of multiple AI models and effort parameters. AI vendors offer multiple model types and specializations, each with their own effort-level variables. Mixing and matching different models serves a similar function as fielding a mixed team of human pentesters — each bringing different specialisms and trained instincts.
• Multiple reruns of autonomous pentests using identical scope and inputs, then collating and rationalizing their combined outputs — reducing the artifacts of per-run randomness.
• Use of existing automated pentest tooling systems and architectures that strictly adhere to proven testing methodologies.
• Human in the Loop (HitL) review and editing of AI-generated pentesting plans to ensure full scope adherence and to identify and remove rogue ambitions before they are executed.
• Human on the Loop (HotL) review of AI findings, audited validation of scope adherence, and manual execution of missed or poorly covered testing paths.
• Reinforcement learning processes to retrain and augment AI models using data from human-delivered pentests.

The Rerun Strategy—and Its Limits

Multiple reruns of autonomous pentest agents are a common practical solution to uncovering a broader spread of vulnerabilities. Four separate autonomous pentest runs may aggregate to uncover 88% of a target's discoverable vulnerabilities, despite each run never independently finding more than 50%.

Each pentest run may follow different testing paths that may or may not intersect or duplicate paths — and consequently findings — of previous runs. In some cases it is advantageous to use different AI models on reruns as a way of increasing path diversity, particularly where certain models are more performant or have been trained differently.

There is a fundamental limitation to the rerun strategy, however. Since it is generally impossible to know in advance how many undiscovered vulnerabilities lie within the target, it is equally impossible to know how many autonomous pentest reruns are required to reach statistically meaningful coverage. You cannot calculate the denominator. This is a variant of the same underlying problem I described in It's Time for Security Analytics to Embrace the Age of Science Over Art — the shift from craft-based, "I ran it and it looks fine" assurance to reproducible, measurable evidence of coverage is hard, and most of the industry hasn't made it yet.

The Enduring Role of Scanners

Modern security scanning technologies remain the only way to efficiently and comprehensively identify vulnerabilities from the 40-year spectrum of previously published and publicly known security flaws. It would be impossible to deliver a pentest with any meaningful assurance of coverage or quality without their use. Both human and AI pentesters are, in principle, equally capable of finding a similar spectrum of vulnerabilities — but neither is as fast or as systematically thorough as a purpose-built scanner for the known-knowns.

This is actually an inconvenient truth about a subset of human pentesters: too many are not significantly better than the scanners and specialist tools they have access to when performing a comprehensive methodology-adherent pentest. They bring genuine value through triaging scanner findings, formulating the specific risk to the customer, and enriching and certifying results in the reporting phase — but the underlying discovery capability is frequently scanner-limited.

The AI pentester naturally outperforms humans in the territory that scanners cover, being immune to grind and fatigue. But AI will likely never be as efficient or as knowledgeable as purpose-built scanner technology for this work. An AI probing a web application for XSS vulnerabilities can do so iteratively and quickly. An AI identifying a three-year-old Cisco router buffer overflow and developing a remote exploit for it blindly is a different proposition entirely — one that a commercial vulnerability scanner resolves in seconds through its built-in signature library and accumulated knowledge base of 50,000 similar device vulnerabilities.

The practical conclusion is that scanners are not a legacy technology being supplanted by AI pentesting. They are complementary infrastructure that both human and autonomous AI pentesters depend on — and should depend on. The SOC world learned a version of this lesson the hard way, and why the SOC needs to learn from the aviation industry's approach to systematic reliability applies equally here: the question is not which single technology is best, but how to layer systems so that the failure modes don't compound. As AI models advance and become more capable, the depth and quality of autonomous testing will improve considerably. But the organizations expecting AI to replace their scanner estate are going to be disappointed. The smarter play is understanding what each layer is actually for.