Note: while there is a difference between a pen test and a vulnerability assessment (vendor risk management often requires both), I will just refer to them as pen tests in the remainder of this post.
Driven by compliance demands from legislators or regulatory bodies, the financial industry and larger companies such as Google or Amazon already implemented pen tests as part of their security program many years ago. This now trickles down to other service providers.
Usually this starts when some of your existing customers ask to fill in their vendor risk management questionnaire or ask similar questions as part of their vendor selection process. Often, there will be a list of open-ended questions about your pen testing practices or some more detailed requirements.
Let’s explore further on how to survive this questionnaire and how crowdsourced penetration testing (bug bounties) will aid you.
The compliance landscape
Banks and other financial institutions must comply with regulatory demands of their National Banks or Monetary Authorities and Basel III requirements. Payment processors and merchants must abide to the Payment Card Industry Standard (PCI). Healthcare organizations in the US must implement the requirements of the Health Insurance Portability and Accountability Act (HIPAA). The US Presidential Residential Policy Directive/PPD-21 provides a framework for a secure and resilient critical infrastructure. Other countries have similar requirements.
The above regulatory requirements are only the tip of the iceberg. Many more security standards, best practices, frame-works and generic cyber-security requirements exist which might be used to provide management or the regulator with assurance that information security risks are managed and controlled.
When those companies use your SaaS platform, you might need to be compliant with the same requirements. Indeed, if important business functionality is running on your platform, this introduces a risk that must be managed. This is done through a process called vendor risk management or third party risk management.
There might be a lot of other requirements to abide by, but let’s focus on pen tests, which are required by most companies that implement a vendor risk management process.
Who is responsible for pen tests?
A third party must execute the pen tests. You cannot declare your own platform or application secure. It is possible and recommended that you run your own internal tests, but at the end, a third party must execute an independent pen test and document the results.
You are required however to follow-up on the pen test results and prove to your customer that you are managing the issues.
Apart from following up on discovered vulnerabilities, your customer might also inquire about your own vendor risk management process with respect to pen test vendors or platforms. Expect to answer questions related to:
-
the expertise of the pen test supplier
-
scope and methodology of the pen test
-
confidentiality of the results
-
confidentiality of customer information accessed during the pen test
-
reporting standards and qualification and prioritization of issues
-
follow-up process including time-lines for corrective action
You might need to provide your customer with the latest pen test report, or at least an executive summary of the results.
How often are pen tests needed?
Standards or regulatory requirements do not often dictate exact timings. In general, auditors will expect you to conduct pen tests at least once a year or when major changes occur. A re-test might also be needed if issues were detected during an earlier test.
Software changes are often delayed and go-live dates are often fluid, making it sometimes hard to plan for a pen test with a regular vendor, who must assure that his ethical hacking team is available when you want the test executed.
Crowdsourced pen testing platforms such a **Cobalt.io** make this easier: a large pool of vetted researchers ensure that pen tests can be executed exactly when you are ready and on short notice.
Expertise of the researchers (ethical hackers)
When you hire a traditional pen test company, the expertise of that company and of the pen testers is often second-guessed by looking at the references or sample projects at their customers.
Curriculum vitae of the individual researchers might give insight on which projects they worked on, how many years of experience they have and which security certifications they hold.
Unfortunately, often times the referenced projects were not done by the researchers included in the proposal and you will not have any guarantee that the same researchers will even work on your application.
Although security certifications do have value, multiple choice questions during an exam might not give any insight on the hacking capability of the researcher. Similarly, being able to penetrate a test environment or finding vulnerabilities in a demo application, does not really teach someone how to hack a real-world application. Out-of-the-box thinking is very difficult to capture in a standardized exam.
A crowdsourced pen testing platform such as Cobalt.io has a large pool of researchers. They earned their stripes and their worth is very visible: the hall-of-fame immediately ranks researchers based on real findings and their ability to create a quality vulnerability report. Individual researchers are listed with their skills and certifications as well as a high-level overview of issues discovered by them. Many are active on social security platforms or have their own blog.
This provides you with a great view of the skills of each individual researcher, second guessing is not needed at all. You can invite each of them to work on your application.
More than one researcher can work on your application, and the differences in skill sets will ensure that the attack surface of your application is fully covered.
Scope and methodology of the pen test
Online pen testing platforms enable you to document your specific requirements, concerns, scope, and focus of the test.
Experienced researchers will often have a private methodology based on the gold standard of OWASP. They are driven by the simple fact that they are not paid unless they find issues. The broader and deeper their research, the more opportunities they have to get paid.
In order to ensure full coverage, an experienced researcher will vet the individual issue reports and ensure that the scope is fully covered. This is part of the standard offering at Cobalt.io.
Confidentiality of results or data accessed
All Cobalt.io researchers have signed a non-disclosure agreement (NDA). This offers the same assurance as a NDA signed as part of a hiring process.
During a pen test, data of customers might be accidentally accessed. Make it clear in the scope description that this is where the pen test stops.
Results of the tests are kept secret. However, it might be part of your marketing plan to be very open about discovered (and fixed) issues. More and more companies are doing this, it proves to their customers that they take security seriously.
As opposed to a regular pen test, a crowdsourced pen testing platform allows you to have a choice:
-
keep the results a secret
-
open-up the results as part of your marketing effort. This requires that both you and the researcher agree on publicizing the issue.
Quality reports and follow-up
Let’s face it: ethical hackers — including myself — spend more time testing than writing a report. But at the end, the report is the only paper trail that enables you to be compliant with the vendor risk management process of your customer.
The Cobalt.io pen testing platform and process ensures that security researchers deliver a quality report:
-
the reported issue is within the scope of the program
-
the issue is well documented, it can be re-executed based on the given information
-
the impact on the application is clear, and a reasonable vulnerability rating is given
-
potential fixes, counter measures or mitigating controls are included
-
further references (what is the issue, how to fix,…) are given.
This online reporting platform enables to give a view on what the status is of a reported vulnerability: reported, accepted or not accepted, ready for re-test, verified fixed. It is immediately clear what the status is.
The auditors of your customers might not go into this level of detail, but will often require a high-level written report. The Cobalt.io platform delivers this written report. This includes:
-
an executive summary
-
an overview of detected issues and classification
-
a link to the OWASP top-10
-
further details and links to the individual detailed reports on the online platform
-
status of re-tests if executed.
TL;DR (Too long, didn’t read)
Conclusion is that online crowdsourced pen testing platforms such as Cobalt.io offer an easy solution to be compliant with the pen test requirements of the vendor risk management process of your customer.
