As 44Con 2012 starts to gain momentum (we’ll be there again this time around) I was perusing some of the talks from last year’s event…

It was a great event with some great presentations, including (if I may say) our own Ian deVilliers’ *Security Application Proxy Pwnage*.  Another presentation that caught my attention was Haroon Meer’s *Penetration Testing considered harmful today*. In this presentation Haroon outlines concerns he has with Penetration Testing and suggests some changes that could be made to the way we test in order to improve the results we get. As you may know a core part of SensePost’s business, and my career for almost 13 years, has been security testing, and so I followed this talk quite closely. The raises some interesting ideas and I felt I’d like to comment on some of the points he was making.

As I understood it, the talk’s hypothesis could be (over) simplified as follows:

1. Despite all efforts the security problem is growing and we’re heading towards a ‘security apocalypse’;
2. Penetration Testing has been presented as a solution to this problem;
3. Penetration Testing doesn’t seem to be working – we’re still  just one 0-day away from being owned – even for our most valuable assets;
4. One of the reasons for this is that we don’t cater for the 0-day, which is a game-changer. 0-day is sometimes overemphasized, but mostly it’s underemphasized, making the value of the test spurious at best;
5. There are some ways in which this can be improved, including the use ‘0-day cards’, which allow the tester to emulate the use of a 0-day on a specific system without needing to actually have one. Think of this like a joker in a game of cards.

To begin with, let’s consider the term “Penetration Testing“, which sits at the core of the hypotheses. This term is widely used to express a number of security testing methodologies and could also be referred to as “attack & penetration”, “ethical hacking”, “vulnerability testing” or “vulnerability assessment”. At SensePost we use the latter term, and the methodology it expresses includes a number of phases of which ‘penetration testing’ – the attempt to actually leverage the vulnerabilities discovered and practically demonstrate their potential impact to the business – is only one. The talk did not specify which specific definition of Penetration Test he was using. However, given the emphasis later in the talk about the significance of the 0-day and ‘owning’ things, I’m assuming he was using the most narrow, technical form of the term. It would seem to me that this already impacts  much of his assertion: There are cases of course where a customer wants us simply to ‘own’ something, or somethings, but most often Penetration Testing is performed within the context of some broader assessment within which many of Haroon’s concerns may already be being addressed.  As the talk pointed out, there are instances where the question is asked “can we breached?”, or “can we be breached without detecting it?”. In such cases a raw “attack and penetration” test can be exactly what’s needed; indeed it’s a model that’s been used by the military for decades. However for the most part penetration testing should only be used as a specific phase in an assessment and to achieve a specific purpose. I believe many services companies, including our own, have already evolved to the point where this is the case.

Next, I’d like to consider the assertion that penetration testing or even security assessment is presented as the “solution” to the security problem. While it’s true that many companies do employ regular testing, amongst our customers it’s most often used as a part of a broader strategy, to achieve a specific purpose. Security Assessment is about learning. Through regular testing, the tester, the assessment team and the customer incrementally understand threats and defenses better. Assumptions and assertions are tested and impacts are demonstrated. To me the talk’s point is like saying that cholesterol testing is being presented as a solution to heart attacks. This seems untrue. Medical testing for a specific condition helps us gauge the likelihood of someone falling victim to a disease. Having understood this, we can apply treatments, change behavior or accept the odds and carry on. Where we have made changes, further testing helps us gauge whether those changes were successful or not. In the same way, security testing delivers a data point that can be used as part of a general security management process. I don’t believe many people are presenting testing as the ‘solution’ to the security problem.

It is fair to say that the entire process within which security testing functions is not having the desired effect; Hence the talk’s reference to a  “security apocalypse”. The failure of security testers to communicate the severity of the situation in language that business can understand surely plays a role here. However, it’s not clear to me that the core of this problem lies with the testing component.

A significant, and interesting component of the talk’s thesis has to do with the role of “0-day” in security and testing. He rightly points out that even a single 0-day in the hands of an attacker can completely change the result of the test and therefore the situation for the attacker. He suggests in his talk that the testing teams who do have 0-day are inclined to over-emphasise those that they have, whilst those who don’t have tend to underemphasize or ignore their impact completely. Reading a bit into what he was saying, you can see the 0-day as a joker in a game of cards. You can play a great game with a great hand but if your opponent has a joker he’s going to smoke you every time. In this the assertion is completely true. The talk goes on to suggest that testers should be granted “0-day cards”, which they can “play” from time to time to be granted access to a particular system and thereby to illustrate more realistically the impact a 0-day can have. I like this idea very much and I’d like to investigate incorporating it into the penetration testing phase for some of our own assessments.

What I struggle to understand however, is why the talk emphasizes the particular ‘joker’ over a number of others that seems apparent to me. For example, why not have a “malicious system administrator card”, a “spear phishing card”, a “backdoor in OTS software” card or a “compromise of upstream provider” card? As the ‘compromise’ of major UK sites like the Register and the Daily Telegraph illustrate there are many factors that could significantly alter the result of an attack but that would typically fall outside the scope of a traditional penetration test. These are attack vectors that fall within the victim’s threat model but are often outside of their reasonable control. Their existence is typically not dealt with during penetration testing, or even assessment, but also cannot be ignored. This doesn’t doesn’t invalidate penetration testing itself, it simply illustrates that testing is not equal to risk management and that risk management also needs to consider factors beyond the client’s direct control.

The solution to this conundrum was touched on in the presentation, albeit very briefly, and it’s “Threat Modeling“. For the last five years I’ve been arguing that system- or enterprise-wide Threat Modeling presents us with the ability to deal with all these unknown factors (and more) and perform technical testing in a manner that’s both broader and more efficient.

The core of the approach I’m proposing is roughly based on the Microsoft methodology and looks as follows:

1. Develop a model of your target environment, incorporating all players, locations, and interfaces. This is done in close collaboration between the client and the tester, thus incorporating both the ‘insider’ and the ‘outsider’ perspective;
2. Enumerate all potential risks, and map them to the model. This results in a very long and comprehensive list of hypothetical risks, which would naturally include the 0-day, but also all the other ‘jokers’ that we discussed above;
3. Sort the list into some order of priority and group similar hypothetical risks together;
4. Perform tests in order of priority where appropriate to prove or disprove the hypothetical risks;
5. Remediate, mitigate, insure or inform as appropriate;
6. Rinse and repeat.

This approach provides a reasonable balance between solid theoretical risk management and aggressive technical testing that addresses all the concerns raised in the talk about the way penetration testing is done today. It also provides the customer with a concrete register of tested risks that can easily be updated from time-to-time and makes sense to both technical and business leaders.

Threat Modeling makes our testing smarter, broader, more efficient and more relevant and as such is a vital improvement to our risk assessment methodology.

Solving the security problem in total is sadly still going to take a whole lot more work…