Pentesting and DevOps: An Engineer's Perspective

In a healthy organization, engineering and security teams should work in a close and efficient manner. I’ve spent years managing engineering organizations, and I have seen instances of both a fruitful relationship and a work in progress.

Coming to Cobalt, I now have two perspectives on the issue: my own, based on my team’s interactions with Ray Espinoza and his very capable security team, and that of the customers we work with daily. Those customers want efficient, impactful pentests — done fast, to a high standard, and built to align with today’s rapid development cycles.

Is such a thing even possible?

Here’s the challenge, as I see it:

DevOps practices require speed, automation, and agility. Traditional penetration testing services rarely focus on these points, making it difficult for engineers to prioritize development and remediation simultaneously.

In this article, I’ll share four specific challenges pentesting brings to DevOps and how a Pentest as a Service (PtaaS) model can help to overcome them. I’ll draw on my own real-world experience and pull in some anecdotes from our customers and other practitioners on the frontlines of this challenge.

The Push Towards DevSecOps

DevSecOps is about integrating security practices into engineering workflows — not as an ‘add-on,’ but as an engineering-led practice that guides code development. Ideally, engineering teams should learn how to own application security. The good news is that security teams are here to guide, train, and consult us during that process. That’s how we do it at Cobalt, and that’s how I believe it works best.

For DevSecOps to be possible, security and engineering teams must understand each other’s goals.

For engineering teams, these usually are:

• Delivering new features at scale and ensuring their quality and reliability.
• Meeting compliance requirements and achieving any certifications important to the business, e.g., ISO 27001 or SOC 2.
• Implementing and maintaining good practices from the start that remove (or better yet prevent) tech debt. Consider what Denali Lumma, VP of Engineering at Project Rōnin, had to share at our AppSec conference on this point:

“At a certain point, it’s too late. There’s too much tech debt, too much complexity, the codebase is too big, and it becomes impossible to do anything about it. What you see there is the business will slowly die, because shipping features becomes harder and harder, critical vulnerabilities and bugs are introduced more frequently, and eventually, it will kill the business.”

And last, but certainly not least, we aim to prevent a security breach. Aside from damaging the organization’s reputation and bottom line, a security breach cripples the engineering team’s ability to execute its roadmap. Instead of working on new features, entire teams must commit to fixing the problem, sometimes for months — time they could invest more profitably elsewhere. Frozen roadmaps can spell the death of an entire organization in the face of competition and stakeholder expectations.

To reach these objectives, especially when following DevOps principles, engineers need processes that are:

• Automated
• Quick
• Agile

I can tell you that the teams I’ve helped lead (including the current one) were all laser-focused on execution and results. We needed to meet aggressive timelines while upholding a high standard of excellence and being strategic about investing talent and resources. Anything that slows us down or impedes our ability to deliver is a threat.

To improve their organization’s DevSecOps maturity, security leaders should suggest workflows that align with those three adjectives.

4 Challenges to Overcome

As valuable as pentesting is, several things stand in the way of successful integration into the DevOps workflow.

Here’s my view of where these challenges lie, plus an actionable solution to each one.

Challenge #1: Strategy

The first step is to agree on a strategy with a clear set of metrics that both teams buy into, along with an agreed plan of how they should improve over a 1-2 year period.

Solution: Add a pentesting section to the engineering team’s existing system health metrics, with KPIs taken directly from each pentest report.

They could include:

• Number of vulnerabilities found per pentest
• % of findings fixed within one quarter
• Time spent fixing critical vulnerabilities
• Historical vulnerability trend (overall and per asset) — an increasing trend should prompt further investigation and a root cause analysis

Consider how you can achieve incremental improvements over time. It’s important to collect enough data to see trends and measure your success towards the set goals.

A tip from me: it will be more effective to move away from one-off testing and implement a long-term series of pentests that consistently supply structured data and allow for more thorough assessments.

“We want to get to a clear picture of reality that we can share with ourselves and with the leadership team, so we know what to expect, and we know if we’re improving or not.” — Denali Lumma, VP of Engineering, Project Rōnin.

Challenge #2: Integration

Historically, engineering teams received pentest results in PDF format with minimal context or remediation guidance.

This approach simply isn’t realistic. It leads to underwhelming results because it forces the use of cumbersome manual ticketing. Opening a ticket for each issue with zero automation is a tedious process. It’s in direct opposition to DevOps, automation, and all the benefits that come with them.

Solution: A PtaaS platform can integrate pentest findings with existing processes and technologies such as Jira or GitHub. With bi-directional integration, engineers receive tickets in their backlogs and automatically trigger a retest by changing the tickets’ status.

“One of the things I like a lot about the platform is once an effort gets underway, I know it’s gotten underway because my inbox starts filling up with messages of vulnerabilities found.” — Chris Wallace, Principal Security Architect, Vonage.

Challenge #3: Guidance

Most engineers aren’t security experts, nor should we expect them to be. They won’t necessarily understand all reported issues, how to reproduce them locally, or what remediation steps to take. Security teams could add huge value here by providing expertise and guidance whenever required.

Solution: Create open communication channels between the teams involved in testing — engineers, security, and pentesters. Tools that allow easy multi-party communication will work, but a PtaaS platform has the added benefit of storing chats, findings, and reports in one place.

“What my engineering team liked about this [PtaaS] engagement is they had the opportunity to discuss issues with pentesters and review the priorities together, instead of having all these findings dropped on them.” — Sergey Stelmakh, Platform Security Architect, MuleSoft.

Challenge #4: Recurrence

Because of the way vendors have traditionally delivered pentesting, many organizations still opt for large, infrequent tests. In a DevOps environment, this approach falls short.

My time at Cobalt has allowed me to witness the inner workings of more mature security organizations. One of the common threads is the use of regular pentests to identify and remediate vulnerabilities systematically.

Product releases are no longer an infrequent occurrence. Many modern engineering teams develop new features and product releases on a weekly basis. As valuable as pentests are, holding a large pentest once or twice per year doesn’t provide the current security insights needed to improve product security.

Solution: Smaller, faster, and more frequent pentests are far more suited to DevOps workflows. This approach ensures a constant stream of security feedback, which is easier for engineers to address than large, infrequent batches of results.

Ideally, pentests should be implemented on-demand rather than to a rigid schedule, as engineering milestones like new feature deployments can happen at irregular intervals. This approach to pentesting meshes well with multi-year pentest programs and PtaaS platforms, both of which support frequent, on-demand testing.

“We release new features every week, and the product one year from now and today are two different products. The pentest report from last year is not relevant anymore.” — Igor Andriushchenko, Director of Quality and Security, Snow Software.

How To Embrace the DevSecOps Challenge

Building pentesting and other security practices into engineering workflows can be a significant cultural and technical challenge. The high-level overview of what engineers need from the security team comes down to this:

• Define and agree on strategy and metrics, so both teams know what they’re aiming towards.
• Find solutions that integrate with existing software development tools and allow automated information sharing flows, e.g., pentest findings and updates.
• Provide a dedicated space where all involved parties can discuss, collaborate, and prioritize.
• Build well-defined but flexible processes where pentest scheduling can be adjusted to roadmap changes.

We all know change is hard. However, with engineering teams pushing new features and product updates faster than ever, it’s an essential precaution to take if you want to avoid accumulating tech debt.

For more tips on aligning pentesting with DevOps, check out our latest guidebook with Larry Maccherone. Also, see how other companies benefit from the flexible pentesting solutions offered at Cobalt with the Aircall case study.