Learn how Cobalt’s Pentest as a Service (PtaaS) model makes you faster, better, and more efficient.

Q&A: Crowdsourced Pen Testing 101

We recently launched our new webinar series, AppSec Disrupted. During our first session, we talked about crowdsourced security and how it…

We recently launched our new webinar series, AppSec Disrupted. During our first session, we talked about crowdsourced security and how it is disrupting the application security landscape. We learned about what a crowdsourced pen test is and how it’s different from a more traditional pen test.

If you’re interested in viewing the on-demand webinar recording, click here.

We saved some time at the end of the webinar for Q&A, but we didn’t get to answer all the questions that were submitted. Here they are!

Q: What’s the difference between a crowdsourced bug bounty and a crowdsourced pen test?

A: A public bug bounty is continuous, and researchers are paid a bounty for being the first to find a bug. In this model, you might have thousands of security researchers looking at an application, or you might have zero. You really have no idea. Some of them may be very good, while others might be…somewhat new to the game. With so many researchers engaged, a public bug bounty tends to produce a lot of duplicate reports which must be triaged (typically an organization must hire 1 or more FTEs just to filter out all the noise). Each security researcher tends to bring with them their “bag of tricks” which they will opportunistically try out to see if they can get paid a bounty, then move on. For an organization that runs a public bug bounty, if there are no reports submitted about an application’s API, there is no way to tell if there are no security bugs to be found, or if no one even looked. There’s not much incentive for a bounty hunger to dig deep into an app, since he or she will only be paid for finding an issue before another researcher beats them to it. The findings produced in a public bug bounty can be described as “scattered.”

A crowdsourced pen test is fixed-price, for a fixed-time period. A purpose built team is selected by matching their skills with the specific technology stack in scope for the engagement. They follow a methodology (for example, OWASP Top 10 and ASVS for web apps), exploring the complete application. The pen test lead is responsible for reviewing each report before it is submitted to ensure the report is valid.

Q: What kind of differences do you see in companies that choose to do pen testing internally, perhaps by investing in pen test tools, versus outsourcing pen testing to a third party, such as in a crowdsourced pen test?

A: The BSIMM shows that 57% of their participants use internal pen test tools, whereas 86% use external penetration testers to find problems.

The advantage of hiring an internal penetration tester is that an FTE can develop a deep knowledge and understanding of an application. The hard part is finding strong talent to hire (according to the CyberSecurity Jobs Report, the workforce shortage is expected to reach 1.5M by the year 2019). Additionally, any specific individual only knows as much as he or she knows.

The advantage to hiring a third party penetration tester (whether traditional or crowdsourced) is that an organization can get a different set of eyes on their application. Professional pen testers can be extremely skilled at what they do, and different people can think creatively to test applications in new ways. Many organizations have a strategy where they rotate different sets of traditional pen testers; others use crowdsourced pen testing and request different researchers to participate in pen test teams each time.

Q: How will a crowdsourced pen tester understand the business context of an application in order to identify business logic flaws? How do they assign criticality ratings to findings? Are they just based on standard CVE ratings?

A: A pentest kicks off with a meeting between the lead researcher and the application’s owners and developers. This discussion covers topics like verifying the scope of the testing, explaining key features and data flows, and ensuring test accounts are in place. It’s also a chance to talk about some high-level threat models in order to help shape the pen test and make it more effective.

The pen test team uses this knowledge to assign a risk score to each finding based on probability of occurrence and business impact.

Q: Do crowdsourced pen testers work as a team on a particular project? How is the scope divided among the pen testers?

A: A team usually consists of a pen test lead and a couple of technical domain experts. They work together as a team to explore the complete application by systematically reviewing its features and components over a fixed time period. They work collaboratively to perform manual security testing related to topics like input validation, authentication, and access controls in order to identify flaws in the application’s implementation. A typical pen test lead might have 16 years of professional experience, and it’s up to him or her to decide how to divvy up the testing.

Q: How do you source your crowd of pen testers?

A: While thousands of people might want to become a crowdsourced pen tester, less than 5 percent of applicants are accepted. To even be considered, a security researcher must be recommended by someone who is already in the community. The vetting process also includes third party government ID verification, social media account review, and a thorough interview over video conference call.

Each crowdsourced pen tester has a profile that displays his or her skills, experience, and performance (as scored by team members and clients). Anonymous testers are not allowed.

Next month, join us for the second session in the AppSec Disrupted Series, Software Development Lifecycle #FAILS.

Check out other posts by Cobalt If you want to learn more about crowdsourced pentesting.

Back to Blog
About Caroline Wong
Caroline Wong is the Chief Strategy Officer at Cobalt. As CSO, Caroline leads the Security, Community, and People teams at Cobalt. She brings a proven background in communications, cybersecurity, and experience delivering global programs to the role. More By Caroline Wong