State of Pentesting 2024: The Impact of AI and LLMs on Penetration Testing

We are thrilled to present the sixth edition of our annual research report, The State of Pentesting 2024.

With over 4,000 pentests, this year’s report explores the transformative impact of Artificial Intelligence (AI) and Large Language Models (LLMs) on the field of penetration testing. The 2024 report uses pentest data and survey results from over 900 responses from security practitioners in the United States and the United Kingdom.

Key insights from the 2024 report include:

• Adoption of AI tools: A significant 75% of SoPR 2024 respondents said their team has adopted new AI tools.
• Top vulnerabilities: The top three vulnerability types that come up regularly in pentests of AI-driven tools include Prompt injection, model denial of service, and prompt leaking.
• Demand for AI: A notable 57% of respondents say the demand for AI has outpaced their security team’s ability to keep up.

How AI is Transforming Pentesting

Penetration testing or pentesting is a crucial aspect of cybersecurity. It involves simulating cyberattacks on an organization's network or applications to discover vulnerabilities and assess their potential impact. 

The advancement of AI has started to transform the pentesting process forcing the industry to balance the risks and rewards posed by this new technology. Let's take a closer look at how AI has begun to transform different stages of the pentesting process.

AI Applications Need Pentesting, Too!

This year’s State of Pentesting Report 2024 demonstrates that information security professionals across the board are seeing significantly increased use of AI tools, and that they’re concerned about being able to manage the associated risk appropriately.

A staggering 92% of U.S.-based developers are integrating AI tools into their workflows, leveraging these technologies with an expectation of enhanced code quality, reduced incident resolution times, and accelerated development cycles.

At Cobalt, we’ve seen a dramatic increase in the number of requests for manual pentests which target AI applications, such as chatbots. The types of vulnerabilities found are sometimes similar to those found in more traditional pentest engagements (denial of service, sensitive data exposure), but some of them are fairly specific to these new technologies (prompt injection, including jailbreaking). Another type of LLM specific finding has to do with the ease with which an attacker might be able to bias the data and impact the results of user queries. This type of attack, known as data poisoning, becomes challenging to detect since attackers manipulate the behavior of these deep learning systems.

Beyond testing AI-enabled applications, the progression of this technology also opens avenues for improvements to existing pentest workflows.

AI Transforms Pentester Focus

Every comprehensive manual penetration test involves two key components: 

1. Performing testing
2. Writing a pentest report 

Most pentesters, by nature, prefer to spend as much time and energy as they can on performing testing which will reveal security vulnerabilities that can be exploited. Writing pentest reports is necessary but not always the most enjoyable aspect of the work that needs to be done. 

This is a perfect fit for AI, which Cobalt pentesters are leveraging to support their report writing. This enables pentesters to spend more time looking for security vulnerabilities, which is their favorite aspect of the engagement. Win-win!

Remediation Efficiency

Collaboration between security professionals, developers, and pentesters is absolutely critical to effective remediation of security vulnerabilities. Developers are the folks who know how apps are supposed to work and how workflows operate. Pentesters can find bugs and also re-test them when the issues are fixed. Security professionals play an important role in connecting these groups in order to get work prioritized and executed upon. 

As AI quickly evolves, we can easily imagine a future state where it’s being used to help organizations to de-duplicate and prioritize their security vulnerability findings from a number of different defect discovery processes. 

It can seem overwhelming if a development team is encountering a large amount of security vulnerabilities that need to be fixed. Where there may be an option to perform one fix and eliminate multiple vulnerabilities, that approach is certainly welcome. Today this type of filtering and association of different vulnerability instances to each other may be a highly manual task on the part of the security team, however this is a ripe opportunity for AI to come in and make a difference.

Embracing the Future of Pentesting with Caution

The advent of AI and LLMs has undeniably changed the pentesting landscape. From the applications enabled with AI to the security workflows improved with this technology, we see the promise of an AI revolution coming closer to reality within the world of cybersecurity.

As we embrace these advancements, it’s crucial to be aware of the potential risks and take proactive measures to mitigate them. For a more comprehensive understanding of the state of pentesting in 2024, download the full State of Pentesting Report 2024.