Leveraging Security Metrics To Protect Your Network

Maybe we should just give up trying to maintain secure enterprise networks; it’s just too hard. Fully 71% of respondents admitted that their networks are exposed to external threats due to misconfiguration issues present in their security device infrastructure. Verizon reports that 79% of organizations fail to maintain their PCI compliance from their prior year’s assessment to the next year’s Initial Report on Compliance. More than 50 percent had no idea how many of their organizations’ internal hosts were actually exposed to the Internet. 

We know that even in this era of constrained budgets, enterprises are spending more and more on network security—and yet 75% of network and security pros agree that the advantage is still on the side of the attacker. Verizon reposts that security “erosion” over the course of the year between PCI audits is the case with the vast majority of enterprises, despite the fact that we know there’s a correlation between data breaches and lack of PCI compliance.

Maybe it’s time to re-evaluate our priorities. As Dr. Mike points out, there’s a general consensus that much can be gained by focusing on the basics—the core controls. If you’re covering 90% of the core controls, security pros agree it’s better to put effort into getting to 100% rather than expanding the number of controls.

But if you’re focused on the core controls, how do you know what percentage level you’re at, and where the areas of exposure are? That’s where security metrics comes in.

In this case, we’re referring to actionable security metrics—metrics that provide proactive security intelligence. Many metrics are available to security pros: number of patches; number of vulnerabilities; and the number of firewall and router config changes are good examples of typical metrics. But most of these data points are without context, or simply serve as busyness measures. They don’t characterize risk in a meaningful way, nor do they point towards a specific area that needs attention.

Andrew Jaquith, in his book Security Metrics: Replacing Fear, Uncertainty and Doubt, describes the value of security metrics by contrasting to other business disciplines. For example, freight companies know their freight cost per mile and loading factors-as well as those of their competitors. Management can therefore set meaningful objectives and measure themselves against comparable companies. Choosing to be above, on, or below an industry average is a question of strategy as well as operational efficiency. For example, a freight company may be willing to have a lower load factor than its peers if that's the tradeoff required to offer faster delivery times (for which it presumably charges a premium).

Similarly, warehousing firms measure and compare their cost/square foot and inventory turns, and e-commerce companies measure their website conversion rates. And of course financial metrics have been standardized and reported on for years. Companies can therefore compare relevant metrics to those of their peers in order to better evaluate their internal performance.

Could such a use of metrics apply to security? Yes, but only if consistently generated within the context of a security framework.

The three pillars of security are visualize, comply and protect. If we build a framework on those pillars we’ll be able to generate meaningful security metrics.

Visualize: There is wisdom in Requirement 1 of the PCI DSS, in the section entitled “Build and Maintain a Secure Network”: the requirement is to create a network diagram, and keep it current. Why? You can’t secure what you can’t see. And yet, according to Verizon Requirement 1 has the second-highest erosion factor out of the nine requirements not specific to planning and checking. When security pros can visualize the network topology—including groups that clearly identify zones (such as DMZ) and untrusted sources—they become much more effective in creating effective segmentation strategies and policies, and maintaining their compliance. 

Comply: Compliance refers to PCI, FINRA, FFIEC, SOX and other regulatory frameworks, of course, but also internal policies, and best practices from sources such as SANS’ 20 Critical Security Controls, Version 3.0. However, complying with regulatory and internal policies in most cases is open loop; we perform security measures in an effort to comply, but other than regulatory audits we’re mostly in the dark as to how effective our security controls are. What we need to do is get from open loop security frameworks to closed loop with feedback controls that allow us to make continuous adjustments in the presence of security erosion, as shown in the diagram below:

Protect: The fundamental security question is whether the network is protected. How can we know what’s working, and where additional focus is required? By developing a security framework that provides security metrics—feedback controls, from which effective remediation strategies to security erosion can be devised. Security metrics enable enterprise to answer questions such as:

• What is my overall level of risk, and how does it compare to yesterday, last week, last month and last year?
• How easily can attackers get in?
• How big is my attack surface?
• How much of my infrastructure is undocumented?
• Are investments and actions paying off?
• Where do we need to improve?
• Are we ready for our next audit?

Note that the questions above relate to actual network security, unlike, say, how many hosts were patched in the last month (busyness measure) or how many vulnerabilities are being scanned for (no context).

Are these good security metrics? Let's look at Andrew Jacquith's definition of a good metric:

1. consistently measured, without subjective criteria;
2. cheap to gather, preferably in an automated way;
3. expressed as a cardinal number or percentage, not with qualitative labels such as high, medium and low;
4. expressed using at least one unit of measure, such as "number of hosts directly exposed"; and
5. contextually specific—relevant enough to decision-makers so that they can take action.

The security metrics provided in RedSeal 5 satisfy all of Jacquith’s criteria for good metrics, enabling RedSeal’s customers to continuously monitor their network through a closed loop process and therefore address problem areas—and in doing so protect their organization’s hosts and other sensitive assets.

Visualizing Security Metrics

This is the third and final post discussing Security Metrics: Replacing Fear, Uncertainty and Doubt by Andrew Jaquith. As I noted, Jaquith makes some intriguing and vital points about the need for "good" metrics and "serious analytic scrutiny" to inform executive decision-making on issues of security, compliance, and risk governance. This is an especially important topic today, with organizations everywhere trying to figure out how to stay secure and improve compliance while cutting their expense budget.

Most organizations, when considering appropriate investment levels to deal with risk, are not lacking for data. But lots of data does not equate to relevant information required for sound decision-making. Jaquith's point is that information in the form of metrics -- good metrics, which he defines -- is lacking in many enterprises.

But once good metrics have been defined, how are they communicated to stakeholders? Jaquith dedicates an entire chapter to visualization. He starts by listing his six design principles for visualization of metrics:

1. It is about the data, not the design (resist urges to "dress up" the data)
2. Just say no to three-dimensional graphics and cutesy chart junk (it obscures your data)
3. Don't go off to meet the wizard (or talking paperclips)
4. Erase, erase, erase (removing tick marks and grid lines results in a crisp chart with few distracting lines)
5. Reconsider Technicolor (default colors are far too saturated, and should be muted. Consider a monochromatic palette)
6. Label honestly and without contortions (pick a meaningful title, label units of measure, don't abbreviate to the point where the meaning is not clear)

Like me, Jaquith is an admirer of Edward Tufte, author of several books about information visualization including the classic The Visual Display of Quantitative Information (1983, Cheshire, CT: Graphics Press). According to Tufte, a key to effective visual displays is understanding the goal of your presentation. In Tufte's own words:

At the heart of quantitative reasoning is a single question: Compared to what? Small multiple designs, multivariate and data bountiful, answer directly by visually enforcing comparisons of changes, of the differences among objects, of the scope of alternatives. For a wide range of problems in data presentation, small multiples are the best design solution.

Hence, we have small multiples as a visualization strategy. Here's an example:

From this display, one can look at different categories (in this case, departments) to view comparative performance over time. Once can readily imagine security/compliance applications for this approach, such as dormant accounts by resource, or excessive access rights by department.

In his book Beautiful Evidence (2006, Cheshire, CT: Graphics Press) Tufte introduces a refinement to this concept called the sparkline, which he defines as "small, intense, simple datawords". The example Tufte uses to explain the sparkline concept is a patient's medical data, taken from Beautiful Evidence:

Besides Tufte's small multiples and sparklines, Jaquith's visualization suggestions include indexed and quartile time series charts, bivariate charts, period-share charts, treemaps, and Pareto charts. The key point is that there's not a single graphic approach that works in all cases; one needs to determine the essence of what is being conveyed. The audience almost always consists of busy people, often executives, who need to have information presented clearly and in context. It doesn't do anyone any good to be able to point out after a security event that the "smoking gun" data had been seen, but it was either lost in the noise of too much data, or its significance was not clear.

P.S. It's not necessarily relevant to this post, but my favorite graphical display of quantitative information is an advertisement for one of Tufte's books that regularly appears in Scientific American and The Economist:

More Security Metrics

Although I wrote about Security Metrics: Replacing Fear, Uncertainty and Doubt by Andrew Jaquith earlier, a single post doesn't do this important topic justice. The key theme as expressed by Jaquith is

...information security is one of the few management disciplines that has yet to submit itself to serious analytic scrutiny.

This lack of analytic scrutiny in the form of security metrics makes risk management especially difficult for executive understanding and guidance, especially when discussing the necessary level of investment required. Executives ideally want their security and compliance metrics to answer the following questions:

• How effective are my security processes?
• Am I better off than I was this time last year?
• How do I compare with my peers?
• Am I spending the right amount of money?
• What are my risk transfer options?

As previously discussed, most functions within an enterprise -- HR, finance, manufacturing, supply chain, call center, e-commerce and operations -- have the ability to measure their performance by tracking key metrics, and comparing with other companies in a peer group. Such metrics share the characteristics of being simple to explain, readily lending themselves to benchmarking, and being consistently and automatically collected.

Without such metrics, we're doomed to reactive rather than proactive risk management. Or, as Jaquith calls it, we're on the hamster wheel of pain:

Here are Jaquith's suggested questions for management when measuring audit and compliance processes and their related investments:

1. How much time and effort are security staff spending on audit-related activities? (Metrics: # regulatory audits completed, time/cost of audit activities)
2. Have audits uncovered serious weaknesses in existing controls? (Metrics: % security compliance reviews with material weaknesses, % key external requirements compliant per external audit)
3. How much time and effort are security staff spending fixing problems uncovered by audits? (Metrics: # pending deficiencies and estimated time/cost to complete, time/cost spent on remediation activities)
4. Have audit activities uncovered problems with controls that would affect customer trust or privacy? (Metric: # pending customer-related deficiencies and estimated time/cost to complete)

Only by employing security metrics and submitting to serious analytic scrutiny can an enterprise get security and compliance risk management off of the hamster wheel of pain and onto a level playing field with other disciplines.

Security Metrics

Andrew Jaquith, in his book Security Metrics: Replacing Fear, Uncertainty and Doubt, describes the value of metrics in general and in doing so identifies one of the key challenges in ensuring system security:

Today's information security battleground is all about entitlements -- who's got them, whether they were granted properly, and how to enforce them.

The book describes how metrics can be applied in managing security systems in general, and in entitlements/access rights in particular. Jaquith, a senior analyst at Forrester, cites examples of how other disciplines and industries use key metrics to compare their operations to peer companies. For example, freight companies know their freight cost per mile and loading factors-as well as those of their competitors. Management can therefore set meaningful objectives and measure themselves against comparable companies. Choosing to be above, on, or below an industry average is a question of strategy as well as operational efficiency. For example, a freight company may be willing to have a lower load factor than its peers if that's the tradeoff required to offer faster delivery times (for which it presumably charges a premium).

Similarly, warehousing firms measure and compare their cost/square foot and inventory turns, and e-commerce companies measure their website conversion rates. And of course financial metrics have been standardized and reported on for years. Companies can therefore compare relevant metrics to those of their peers in order to better evaluate their internal performance.

Could such a use of metrics apply to security? And can metrics be of use in the "entitlements battleground"?

First, let's look at Jacquith's definition of a good metric:

1. consistently measured, without subjective criteria;
2. cheap to gather, preferably in an automated way;
3. expressed as a cardinal number or percentage, not with qualitative labels such as high, medium and low;
4. expressed using at least one unit of measure, such as "defects" or "dormant accounts"; and
5. contextually specific -- relevant enough to decision-makers so that they can take action.

So what about the "information security battleground", namely entitlements and access rights? What metrics are relevant to that? Jaquith lists pertinent questions and the metrics that can guide management actions, for example: Does the organization review employee entitlements? An example metric would be % accounts dormant. (The complete discussion starts on page 117 of Jaquith's book under the heading Ensuring System Security.)

One of the advantages of a multi-tenant SaaS solution is the global statistical perspective that can be provided, which allows customers to compare their performance to that of their peers. By knowing industry averages for key metrics  customers can benchmark their internal performance and security objectives to those of comparable organizations. What better way to arm oneself for the information security battleground known as entitlements management?

The definition and application of security metrics is ongoing. One resource I recommend is Securitymetrics.org, which provides empirical strategies for decision-makers and security practitioners and which includes links to digests, presentations, and handouts from past Metricon Workshops.