Every industry is dealing with the new reality that AI agents and large language models (LLMs) can do things faster than humans without ever taking breaks (see vibe coding). Security as an industry is also being subsumed by AI. This is an active topic of discussion among almost every vendor, practitioner, investor, and analyst. Caleb Sima recently published a great article and associated talk from BSides entitled “Predicting AI’s impact on Security.”
The term “AI security” now comes up in almost every conversation I have, from customer calls to booth conversations at both RSA and BlackHat. I often have to channel my inner Justin Bieber and ask “What do you mean?” When people talk about AI security, they are usually conflating two concepts:
- How to use AI and AI-powered tools to enhance your cybersecurity practices.
- How to make sure the AI that you’re building is itself secure and ultimately safe for users.
Putting the doom-and-gloom of “we’re all going to be replaced by AI” aside for a moment, let’s clarify what we mean when talking about the intersection of cybersecurity and AI.
Using AI for security purposes
Let’s start with the use case of leveraging AI to enhance cybersecurity. When the majority of people in our industry bring up AI in conversation, this is what they’re referring to.
Almost every security company I know is jumping on the AI train. AI is being added to existing products, while new startups are popping up every day that claim to have new AI agents that can do all the things humans do—but faster, while never sleeping.
In application security specifically, we’re seeing things like AI vulnerability discovery, agentic vulnerability triage, and auto-patching (coming soon to a SOC near you!). Most of these tools rely on networks of AI agents to perform all the steps necessary from identifying a vulnerability through finding a fix and creating the corresponding pull requests for developers. The ultimate end we’re barreling towards is autonomous, self-healing systems.
This evolution has its pros and cons, and, depending who you talk to, you’ll get a litany of opinions in response. While we can debate the efficacy and outcome of this approach, we can’t debate the current trend towards AI-powered everything.
In fact, Cobalt is working on ways to leverage AI and automation to deliver more efficient and effective pentests. We’re exploring new methods of equipping our human testers with AI, so they can spend more time exploring new exposures that AI can’t uncover on its own. Along those lines, we recently published our commitment to responsibly leveraging artificial intelligence to enhance efficiency and productivity across every department, ultimately delivering superior value to our customers.
Securing the use of AI and LLM-based applications
The other side of AI security that is discussed less frequently, unfortunately, is securing AI applications and LLMs that you are actively building. If you’re deploying AI products for your customers, there are important questions you should be asking.
These include things like:
- Is our chatbot susceptible to prompt injection and data leakage?
- Do you know if the models we’re using have any inherent bias?
- Do you know if the LLM-based application powering our AI agents have any vulnerabilities?
- Do you know if the infrastructure our chatbot is running on has any misconfigurations?
- Can we reliably detect when a large-language model is infringing on open-source copyrights?
The answers to these questions are going to have a greater impact on your business, at least in the near term, than adopting the latest AI-powered security solution.
The facts about AI and LLM security
You might be thinking: “That sounds like your opinion. Where are the facts?”
Based on our research, 32% of findings in our LLM pentests are high-risk vulnerabilities, a higher proportion of serious findings than web applications, networks, or any other kind of pentest we conduct. Despite this fact, only one-fifth of these serious findings ever gets resolved, underscoring why so many security leaders (45%) say a “strategic pause” is necessary in the use of AI in their businesses.
Because of these findings, AI development is greatly outpacing our ability to secure AI systems. While maintaining security tooling and equipping your teams with the most advanced security technology stack will be beneficial from a risk reduction standpoint, securing the AI you’re building is critical to your continued success as a company.
Focusing on safe and secure AI
OWASP recently released their AI Testing Guide. One of the unique things about AI is having to worry about both safety and security. Insecure AI implementations can lead to data breaches, system manipulation, and critical loss of intellectual property.
Unsafe AI, however, poses a different threat: reputational damage, harm to your users, and a loss of customer trust. At Cobalt, we are using new methodologies and frameworks for testing AI and LLMs based on tried and true security practices like pentesting and red teaming.
While there have been tremendous investments and advancements in fully autonomous pentesting, we’re a long way away (if ever) from replacing the ingenuity and expertise of human pentesters. There is likely a future state where the training data is thorough enough, and models are robust enough, that humans are no longer necessary in the testing process. At best, we’re still a few years away from that.
Based on the number of new vulnerabilities reported each year, the pace of innovation of attackers, and the sheer number of cyber incidents, “a few years” is a lifetime for a software-producing enterprise, and a millennium in the day-to-day work of a security analyst.
If you’re building AI into your products, make sure it’s safe, has human oversight, and is secure for your users. Read the State of LLM Security Report to learn more about securing your AI. If you’d like to discuss more about testing your AI, connect with a member of our team today.
