AI in cybersecurity presents IT teams with formidable new challenges while providing powerful, innovative cybersecurity solutions. On one front, the growing popularity and diversity of artificial intelligence apps opens additional attack vectors and gives hackers efficient tools to exploit them. On the opposing side, AI tools help security teams leverage automation to efficiently detect threats, analyze security risks, mitigate vulnerabilities, and report results.
In this blog, we'll survey the ways AI in cybersecurity is transforming both attack and defense strategies, with a focus on generative AI (genAI).
We'll cover:
- How genAI relates to other types of artificial intelligence such as machine learning (ML), natural language processing (NLP), and large language models (LLMs)
- Why genAI output has increased the frequency, sophistication, and effectiveness of cyberattacks
- What security professionals are doing to turn genAI against attackers and harden defenses
Overview of Different Types of Artificial Intelligence
To understand the role genAI plays in security, it helps to distinguish genAI from the growing variety of AI applications flooding today's market. AI as a niche within computer science has been around since the 1940s, often under other names, but over the last 20 years, it has matured into a mainstream technology with a growing range of specializations.
The most important ones for understanding the role of AI in cybersecurity include:
- Machine learning
- Natural language processing
- Large language models
- Generative AI
Each of the last three disciplines in this series represents a specialized application of the broader discipline above it. In other words, NLP applies ML, LLM applies NLP, and genAI applies LLM. To better understand these AI models and their relationships, let's start with machine learning.
Machine Learning
As the name implies, ML simulates certain features of human learning, enabling computer programs to "learn" from data without coding updates. To imitate learning, ML programs apply statistical algorithms to input from a data training set. The program seeks a mathematical pattern that matches the data and can serve as a model for making predictions. Then it checks the accuracy of the model's predictions against data from the training set. If the program can find a better match in the data, it adjusts the model until it meets a designated threshold of accuracy.
ML supports applications such as recommendation engines, speech recognition, and computer vision. For example, Netflix has used ML to help predict user viewing preferences.
Natural Language Processing
NLP applies ML techniques to process everyday human language, enabling human users to interact with programs through natural speech or text input without providing input in formal programming languages. To achieve this, NLP simulates the way the human mind associates words, simulating the brain's neural networks by applying advanced statistical techniques such as nonlinear regression. This enables NLP to analyze relationships between associated words and their meanings.
NLP supports tasks such as speech recognition, text-to-speech conversion, and automated summaries of texts. Common applications of NLP include chatbots and digital assistants such as Apple Siri and Amazon Alexa.
Large Language Models
LLMs apply advanced ML techniques and NLP to large amounts of text data in order to generate text mimicking human-generated text. LLMs differ from NLP in their focus on text-based language, use of large data sets, and deployment of more complex, many-layered neural networks. LLMs employ advanced statistical analysis to analyze text input and generate language models that predict the next text string based on prior input. After using ML techniques to fine-tune model predictions, LLMs can generate human-like text from text input prompts.
LLMs serve genAI applications as well as other uses, such as generating AI search engine summaries, translating texts, analyzing customer sentiment, and executing functions based on text input. ChatGPT offers a popular example of an LLM in action.
Generative AI
GenAI applies LLM input to generate content, such as writing, images, music, or videos. GenAI works by harnessing LLMs to train models on text input or other types of input that have been converted into textual units, called tokens, representing multimedia data. For example, data representing audio tone, volume, or pitch can generate audio content, enabling genAI to create music or audio deepfakes.
GenAI can be used for purposes such as responding to customer support tickets, creating marketing content, summarizing texts, or writing code. For example, GitHub Copilot helps developers write and edit code.
The ability of genAI to write and modify code highlights its relevance to security. Hackers can now use genAI to write and run malicious code. At the same time, security teams can put genAI to use building defenses against cybercrime. Let's look at both sides of the picture.
Malicious AI in Cybersecurity: How GenAI Emboldens Cybercriminals
Following the November 2022 launch of ChatGPT, cybercriminals quickly began using genAI to compose phishing emails and messages at scale. Phishing messages increased 4.2% between November 2022 and March 2024, fueled by genAI, according to the genAI security firm SlashNext. This trend continues, with browser security provider Menlo Security reporting that genAI contributed to a 140% increase in browser-based phishing attacks and a 130% increase in zero-hour phishing attacks in 2024 compared to 2023. Meanwhile, criminals have developed other ways to exploit genAI. Below are some of the most common ways cybercriminals are abusing genAI.
Social Engineering
Phishing
GenAI can craft convincing phishing messages for emails, social media, text messaging, web browsers, and other media. Cybercriminals are using genAI to increase both the efficiency and volume of attacks. Generative technology can train phishing models to mimic writing styles of legitimate companies or individuals, making messages sound authentic. GenAI improves phishing efficiency in other ways, such as making messages sound more urgent, repurposing them for multiple channels, or translating them into other languages. This makes it easier for criminals to create phishing messages quickly and scale up attacks.
Some phishing attacks now impersonate popular genAI tools to trick users into divulging personal information. For example, some criminal organizations offer fake genAI tools to help users write resumes, tricking them into providing sensitive details that enable identity theft.
Vishing
Another troublesome trend with the advent of genAI technology comes from vishing attacks. Vishing, short for "voice phishing," is a type of social engineering attack conducted over video calls. Attackers attempt to trick individuals into revealing sensitive information by impersonating individuals from legitimate entities. This tactic has been used by rogue actors like North Korea, attempting to infiltrate a company’s employee base.
Fake Content Creation
In addition to phishing messages, cybercriminals can use generative technology to create fake content that promotes malicious ends. For example, a genAI program can duplicate the look of a legitimate website, creating a fake portal that criminals can use to harvest passwords, personally identifiable information, and other sensitive data. Similarly, a criminal can impersonate a victim's social media account with a duplicate account generated by AI.
Cybercriminals can use genAI to create fake reviews that promote sales of phony products or services. Disinformation generated by AI can serve criminal ends such as damaging a target's reputation, inciting violence, or putting victims in harm's way. During the Los Angeles wildfires in January 2025, social media spread AI-generated images of Hollywood's iconic sign on fire (this did not happen), potentially confusing residents evacuating the area.
Bypassing Security Checks
Criminals increasingly use genAI to bypass security checks. GenAI can predict passwords by studying password databases, fool CAPTCHA systems designed to spot bots, or generate deepfakes of photo IDs and voices to thwart multifactor authentication.
GenAI also can help cybercriminals evade automated security checks. For example, AI-generated malware can create payloads designed to avoid intrusion detection and prevention systems (IDPS). GenAI also can create botnets that dynamically rotate IPs or robocall numbers, avoiding security blocklisting.
Malware and Exploit Development
GenAI empowers more attackers by lowering the barrier to entry to create malicious attacks. GenAI can help attackers script malicious code or write code to exploit known vulnerabilities, with far less knowledge about coding and security than was previously neecessary. This new technology can create new code or improve existing code for efficiency and stealth. For example, genAI can introduce script that helps malware evade detection by antivirus scanners.
When it comes to exploits, a 2024 Cornell study found that the OpenAI GPT-4 LLM can exploit 87% of one-day vulnerabilities, compared to 0% success rate for other LLMs and open-source vulnerability scanners. However, this success rate drops to 7% without a CVE description. The researchers concluded that, as genAI continues to improve, its capability to exploit vulnerabilities will increase. Moreover, genAI can help attackers detect vulnerabilities, such as web applications vulnerable to SQL injection or cross-site scripting (XSS).
Criminalizing LLMs
Hackers are using genAI to create criminal versions of LLMs or put legitimate LLMs to criminal use. In 2023, criminals began offering an LLM called WormGPT, designed as a malicious alternative to ChatGPT lacking any legal or ethical safeguards to prevent improper use. WormGPT soon acquired a bad reputation that put it out of business, but similar tools emerged in its place.
However, investing in developing malicious LLMs requires work, so hackers now prefer to jailbreak existing LLMs rather than invent new ones, according to security provider Trend Micro. Jailbreak-as-a-service (JaaS) providers now offer prompts designed to trick genAI tools like ChatGPT into bypassing their own safeguards. The Open Worldwide Application Security Project (OWASP) now identifies prompt injections designed to jailbreak LLMs as the top genAI security vulnerability in the 2025 OWASP Top 10 for LLM and Gen AI.
Leveraging AI for Cybersecurity: How GenAI Supports Security Operations
Fortunately, genAI can help security teams as well as criminals. Just as criminals can use genAI to study and exploit vulnerabilities, security professionals can deploy generative technology to spot and mitigate attacks. Here are a few ways genAI is enhancing security operations.
Real-time Threat Detection
While traditional security monitoring detects signatures that represent known threats, machine learning enables genAI to detect new patterns representing statistical deviations from normal behavior. This provides a real-time early warning of suspicious behavior that could indicate an attack. For example, financial service providers can instantly spot signs of potential identity theft attempts. AI-powered detection systems also can scan messages and their contents and contexts for phishing attempts or malicious payloads.
Automating Rapid Responses
After detecting a real-time threat, genAI can trigger automated responses to block traffic, deny attackers resources, and implement mitigations. This reduces windows of vulnerability, denies attackers opportunities to escalate, and minimizes damage from security breaches. Using automation to reduce response time also increases the efficiency of human security agents, freeing them up to focus on priority threats requiring immediate attention.
Proactive Risk Analysis
Beyond detecting existing threats, genAI can leverage LLMs to create models that predict potential attack vectors. For example, by studying historical attack patterns, genAI can anticipate hacker tactics, enabling security teams to develop pre-emptive defenses.
Supporting Penetration Testing
GenAI can support penetration testing (pentesting), which simulates cyberattacks in order to identify vulnerabilities and develop mitigations. By using LLM techniques, genAI can rapidly analyze large quantities of data and generate test scenarios. This includes generating scenarios where attackers creatively respond to attempted defenses, more closely simulating real attacks. GenAI models can continuously learn from security data and improve attack simulations and detection capabilities. Together, these capabilities enable security teams to more rapidly deploy pentesting and evaluate results. Additionally, genAI can help pentesters quickly summarize reports of test results to share with decision-makers and stakeholders.
Deploy Pentesting to Strengthen Your Cybersecurity
While genAI can assist cybersecurity teams with remediation, AI-powered security measures must be integrated with human supervision to be effective. Just as genAI tools like ChatGPT can be prone to hallucinations without human checks, an exclusively automated approach to cybersecurity fails to detect vulnerabilities that evade AI. Rather than replacing human security personnel, genAI can support security teams by automating routine tasks such as threat scanning, and give team members more time to focus on activities requiring human attention.
Penetration testing services with the Cobalt Offensive Security Platform make security testing fast, easy, and reliable. Our team of pentesters works with today's industry leaders to develop standards for fighting today's cyberthreats. We offer on-demand pentesting services–start a pentest in as little as 24-hours–for applications, networks, and AI Penetration Testing Services for LLM Applications specifically designed to identify and mitigate vulnerabilities in AI tools.
