Penetration testing is a critical cybersecurity and compliance tool today, but it's also highly misunderstood. First, pen tests have materially changed in the last couple of years, and many CIOs and CISOs still think of pen tests the way they used to be. Secondly, there are such different implementations and uses of pen tests that the term can be more confusing than illuminating.
The differing opinions begin with the best ways to leverage pen testing. Should it be used broadly to identify a wide range of security holes and improperly retained sensitive data, including problems that neither Security nor IT had even considered? Or is it best used far more narrowly, to look for the existence of previously known hole types, possibly even via solely automated testing? Or is it primarily to identify weaknesses before an attacker does?
The biggest strategic issue speaks to cybersecurity strategies well beyond pen testing: What should an enterprise do with the results? Some argue that it should be used solely to identify problems that they are then quickly negated, while others argue that pen test results are best used as a roadmap to the root causes of the discovered holes. In other words, the results should give hints about how to fix a much more fundamental problem.
It's also critical to explore using pen testing anywhere and everywhere where any company data is touched. That means IoT/IIoT, cloud environments, hosted apps, partner environments (contracts permitting), and even modern fleet vehicles, which are quietly retaining a tremendous amount of data.
"CISOs should use pen testing to match what they perceive as their risk assessment," said Matt Miller, principal security engineer for Triaxiom Security. "What are the top threats they are fighting? What are the risks that they are worried about the most?"
When dealing with a tactic that is as versatile as pen testing, it's important to define a scope, even just a preliminary initial scope.
"It's an issue of prescriptive versus non-prescriptive," said Taylor Smith, a penetration tester at Pivot Point Security. "We've worked with businesses that are trying to tackle compliance but don't know where to start."
Smith argues that pen testing, despite it being a technique that has been used for decades, is often given insufficient consideration by many CISOs.
"Pen testing falls into a specialization that can sit beyond a CISO's typical reach. Pen testing is valuable because it serves as both an adversary simulation as well as a pulse of the current state of assets like networks and web apps and beyond," Smith said. "I think CISOs often misunderstand how this data can be useful and may miss the forest for the trees by focusing on specific results rather than looking long term, seeking what caused those results."
Complicating this is the fact that pen tests today are literally not what they used to be.
"Pen tests from the early 2010s were different than a pen test performed today, and the definitions have become even more vague and flexible," Smith said. "CISOs mean well when they pursue a pen test, but I think their tenure may have memories of very different services from a time of security that has passed."
When strategizing a pen test approach, one of the considerations is identifying who will perform the tests: internal or external talent. One of the top issues to resolve on that can go both ways. Internal talent are often far more familiar with existing systems, but that is both a plus and a minus. That internal expertise can sometimes blind a tester from seeing things, in the same way that writers can often miss their own typos because their brain knows what they were trying to say.
A third-party tester can also sidestep corporate political issues. Let's say that an internal tester discovers a flaw due to sloppy coding. That code may have been written by a specific manager that the internal tester needs a favor from next week. That conflict of interest would likely not impact a third-party tester.
On the flip side, that internal tester is going to likely have an easier time tracking down a root cause because that internal tester knows the environment's history.
"Internal talent and familiarity with systems can be excellent for digging deeply into potential issues and having a constant vigil on the state of security," Smith said. "But the third-party element can help simulate realistic threats and identify shortfalls that internal resources may not be able to. Ideally, the inside and the outside working in tandem is ideal, but not always realistic concerning budget or execution."
Tony UcedaVelez, co-creator of the Process for Attack Simulation and Threat Analysis (PASTA) and the CEO of VerSprite, echoes the idea of trying to combine internal and external.
"Both are definitely important, but using external resources can bring a lot of unbiased expertise to be shared with your internal groups. Teaming up a vendor with your blue or red teams can definitely bring a lot of value in addition to results of the exercise being performed in the form of a learning and consulting opportunity," UcedaVelez said. "We are seeing more and more purple teaming efforts being demanded because of this reason."
Combining automation and pen testing is another dicey issue. Unlike the internal versus third-party issue, automation is almost universally embraced as a powerful—and quite cost-effective—supplement to manual human-driven pen testing. The problem is when CISOs and CIOs try to use automated as the sole means of pen testing.
Smith's position is that automation in pen testing must be encouraged, but sharply limited.
"Automated tools are a huge boon for penetration testers. I don't know any professional who doesn't leverage them. The time saved doing scans is incredible and has evolved even just over the last few years. But there is definitely a line, and I would draw it very aggressively. We are far, far away from the concept of a meaningful automated pen test,” Smith said. "The manual element is crucial to really peeling back the layers of discovery and exploitation in ways that scripts and scanners and automated exploit tasks simply can't. Being able to tailor to the situation, adapt, grow knowledge very quickly and on the fly, those are all things that I've yet to see an automated tool really perform well without the human element."
UcedaVelez agrees, but sees automation as a fine start to the process.
"Automated tools are good for the initial stages of a pentest, recon, discovery, and vuln assessment that help ensure breadth of coverage. But manual testing efforts are best for testing specific abuse cases such as those affecting application logic or context-based business flows," he said. "Manual also works best for exploitation and post-exploitation tasks that can show the real impact of the findings and how they substantiate the attack patterns analyzed."
At its simplest level, an automated pen test can only find what it has been programmed to find. A human pen tester can look around and explore what looks off. Automation will rarely—not never, but rarely—find a hole that no one anticipated.
"One of the psychological challenges for pentesting—especially for enterprise CISOs—also happens to be arguably its greatest strength. It is the fact that pen testing, done properly, is proactive, not reactive," said Tom Brennan, Executive Director, CREST Americas Region. "Given how stressed and overworked security departments are today, CISOs can barely keep up with reactive tasks, putting out fires that seem to crop up hourly. To ask them to be proactive and to actively seek out problems that haven't blow up yet, that can seem very unattractive. But it's precisely what they need to do. They need to get ahead of attacks and to fix holes and backdoors before the bad guys discover them."
CREST has programs to support pen testing, including guidance for commercially reasonable assurance activity for defensible pen tests, as well as a comprehensive guide for running an effective penetration testing program.
"The Guide presents a useful overview of the key concepts you will need to understand to conduct well-managed penetration tests, explaining what a penetration test is (and is not), outlining its strengths and limitations," CREST writes, "and describing why an organization would typically choose to employ an external provider of penetration testing services to help them plan for and undertake tests effectively, ensuing that vulnerabilities are identified and remediated."
