Leveraging Security Metrics To Protect Your Network

Maybe we should just give up trying to maintain secure enterprise networks; it’s just too hard. Fully 71% of respondents admitted that their networks are exposed to external threats due to misconfiguration issues present in their security device infrastructure. Verizon reports that 79% of organizations fail to maintain their PCI compliance from their prior year’s assessment to the next year’s Initial Report on Compliance. More than 50 percent had no idea how many of their organizations’ internal hosts were actually exposed to the Internet. 

We know that even in this era of constrained budgets, enterprises are spending more and more on network security—and yet 75% of network and security pros agree that the advantage is still on the side of the attacker. Verizon reposts that security “erosion” over the course of the year between PCI audits is the case with the vast majority of enterprises, despite the fact that we know there’s a correlation between data breaches and lack of PCI compliance.

Maybe it’s time to re-evaluate our priorities. As Dr. Mike points out, there’s a general consensus that much can be gained by focusing on the basics—the core controls. If you’re covering 90% of the core controls, security pros agree it’s better to put effort into getting to 100% rather than expanding the number of controls.

But if you’re focused on the core controls, how do you know what percentage level you’re at, and where the areas of exposure are? That’s where security metrics comes in.

In this case, we’re referring to actionable security metrics—metrics that provide proactive security intelligence. Many metrics are available to security pros: number of patches; number of vulnerabilities; and the number of firewall and router config changes are good examples of typical metrics. But most of these data points are without context, or simply serve as busyness measures. They don’t characterize risk in a meaningful way, nor do they point towards a specific area that needs attention.

Andrew Jaquith, in his book Security Metrics: Replacing Fear, Uncertainty and Doubt, describes the value of security metrics by contrasting to other business disciplines. For example, freight companies know their freight cost per mile and loading factors-as well as those of their competitors. Management can therefore set meaningful objectives and measure themselves against comparable companies. Choosing to be above, on, or below an industry average is a question of strategy as well as operational efficiency. For example, a freight company may be willing to have a lower load factor than its peers if that's the tradeoff required to offer faster delivery times (for which it presumably charges a premium).

Similarly, warehousing firms measure and compare their cost/square foot and inventory turns, and e-commerce companies measure their website conversion rates. And of course financial metrics have been standardized and reported on for years. Companies can therefore compare relevant metrics to those of their peers in order to better evaluate their internal performance.

Could such a use of metrics apply to security? Yes, but only if consistently generated within the context of a security framework.

The three pillars of security are visualize, comply and protect. If we build a framework on those pillars we’ll be able to generate meaningful security metrics.

Visualize: There is wisdom in Requirement 1 of the PCI DSS, in the section entitled “Build and Maintain a Secure Network”: the requirement is to create a network diagram, and keep it current. Why? You can’t secure what you can’t see. And yet, according to Verizon Requirement 1 has the second-highest erosion factor out of the nine requirements not specific to planning and checking. When security pros can visualize the network topology—including groups that clearly identify zones (such as DMZ) and untrusted sources—they become much more effective in creating effective segmentation strategies and policies, and maintaining their compliance. 

Comply: Compliance refers to PCI, FINRA, FFIEC, SOX and other regulatory frameworks, of course, but also internal policies, and best practices from sources such as SANS’ 20 Critical Security Controls, Version 3.0. However, complying with regulatory and internal policies in most cases is open loop; we perform security measures in an effort to comply, but other than regulatory audits we’re mostly in the dark as to how effective our security controls are. What we need to do is get from open loop security frameworks to closed loop with feedback controls that allow us to make continuous adjustments in the presence of security erosion, as shown in the diagram below:

Protect: The fundamental security question is whether the network is protected. How can we know what’s working, and where additional focus is required? By developing a security framework that provides security metrics—feedback controls, from which effective remediation strategies to security erosion can be devised. Security metrics enable enterprise to answer questions such as:

• What is my overall level of risk, and how does it compare to yesterday, last week, last month and last year?
• How easily can attackers get in?
• How big is my attack surface?
• How much of my infrastructure is undocumented?
• Are investments and actions paying off?
• Where do we need to improve?
• Are we ready for our next audit?

Note that the questions above relate to actual network security, unlike, say, how many hosts were patched in the last month (busyness measure) or how many vulnerabilities are being scanned for (no context).

Are these good security metrics? Let's look at Andrew Jacquith's definition of a good metric:

1. consistently measured, without subjective criteria;
2. cheap to gather, preferably in an automated way;
3. expressed as a cardinal number or percentage, not with qualitative labels such as high, medium and low;
4. expressed using at least one unit of measure, such as "number of hosts directly exposed"; and
5. contextually specific—relevant enough to decision-makers so that they can take action.

The security metrics provided in RedSeal 5 satisfy all of Jacquith’s criteria for good metrics, enabling RedSeal’s customers to continuously monitor their network through a closed loop process and therefore address problem areas—and in doing so protect their organization’s hosts and other sensitive assets.

"Night Dragon" Latest Reported Advanced Persistent Threat

Advanced persistent threats, when detected, are rarely publicly reported. Government agencies and enterprises that may have had sensitive data exfiltrated are reluctant to admit it, even more since they are unlikely to know precisely what assets were stolen. That's what makes McAfee's announcement of the so-called Night Dragon exploit noteworthy.

It's been a year since McAfee aired details of Operation Aurora, an advanced persistent threat (APT) that targeted at least 30 companies and organizations -- notably including Google, who publicly linked the exploit to China.

George Kurtz, CTO at McAfee, writes in his blog:

Starting in November 2009, covert cyberattacks were launched against several global oil, energy, and petrochemical companies. The attackers targeted proprietary operations and project-financing information on oil and gas field bids and operations. This information is highly sensitive and can make or break multibillion dollar deals in this extremely competitive industry.

McAfee has identified the tools, techniques, and network activities used in these attacks, which continue on to this day. These attacks have involved an elaborate mix of hacking techniques including social engineering, spear-phishing, Windows exploits, Active Directory compromises, and the use of remote administration tools (RATs).

 McAfee provided the graphic below to outline the stages of the attack:

The data accessed by the attackers included operational oil and gas field production systems, financial documents related to field exploration and bidding, and data from SCADA systems.

No one knows how many additional exploits are silently underway, exfiltrating sensitive data, intellectual property and state secrets. What's clear is that the current generation of tools to detect and defend against such attacks are inadequate for preventing such breeches.

Controlling Excessive Entitlements

Deloitte, in their 2010 Financial Services Global Security Study, reports that excessive entitlements, also known as excessive access rights, was the top audit finding over the past year -- for the third year in a row. It's not an isolated issue: according to Deloitte, excessive entitlements was the top audit finding  in retail and commercial banking, insurance, investment banking, and globally across all financial service segments.

Since all major regulatory frameworks, including SOX, PCI DSS, GLBA, NERC and HIPAA, require entitlement controls, many thousands of companies globally are obligated to prevent excessive entitlements and yet, according to the Deloitte survey, have failed to effectively do so.

IDC states that up to 60% of entitlements on most systems are expired and therefore dormant. It's no wonder that auditors can readily uncover excessive entitlements.

Contrast that with entitlements managed by online billing systems, where typically 0% of entitlements are dormant. What's the difference? Why are billing systems able to manage their entitlements effectively, while enterprise IT departments cannot?

The answer? Money.

Billing systems turn entitlements on or off based on payment activity. If an end user stops paying for any reason, the billing system notifies the client company and the associated product or service is no longer made available. If it were not so, the company would lose money by providing products or services for which there is no associated revenue -- in other words, operating at a loss. Because they have a financial incentive to get it right, these companies manage entitlements effectively.

Now consider financial services enterprises. When users are transfered from one department to another, or are assigned new roles in the company, they often retain their legacy entitlements through a transition period for support and training purposes. It's safer to keep these entitlements in case questions come up regarding the prior role. But no real incentive exists for end users to later relinquish their now excessive entitlements, and these entitlements often fall through the cracks of IT and compliance tracking systems. An enterprise may spend hundreds of thousands if not millions of dollars on entitlement management systems. But with up to 60% of accounts in the dormant state, the challenge is simply too great without having line-of-business managers and IT staff spend an unreasonable amount of time trying to stay on top of the issue. As a result, most enterprises have found that effectively managing entitlements and access controls is simply not possible.

Financial incentives work, as demonstrated by online billing systems. So why not try that approach in large enterprises? Considering the risk to the business from failed audits, it's time to think outside of the box. So here's an idea:

What if every user had a payroll deduction for every entitlement that is unused for a certain period, let's say 60 days. The "fine" amount goes into a reserved account, and is refunded once the entitlement is relinquished. This establishes a gentle but real incentive for end users -- not IT, not the compliance group, and certainly not HR -- to manage entitlements. By putting the issue into the hands of the only people who know whether their entitlements are required or not to perform their job functions, and underlining it with a mechanism to ensure visibility and remediation, the problem of excessive entitlements could be solved once and for all.

Advanced Persistent Threats

Security trends tend to focus on technology: terminology such as malware (on the exploit side) and data leakage protection (on the security solutions side) describe the issue in terms of their most salient technical characteristics. Botnets, drive-by downloads, and Trojan horses add further color to the technical aspects of key security threats.

The “who” behind these security threats is generally thought to be less interesting. Yes, we think we know that certain botnets are controlled by the Russian mafia, and certain exploits tend to be perpetrated by insiders. But it’s the technology behind these threats that we in the high-tech security business use to identify them and their remediation.

Advanced Persistent Threats, recently made trendy by security vendors’ marketing departments, seem fundamentally different. If you look to technical descriptions of advanced, persistent threats (APTs) you will have trouble distinguishing them from botnets. FireEye, for example, describes various command and control systems that bots and APT have in common. Shared characteristics between botnets and APT include stealth, polymorphism (continuously altering malware as it goes from host to host), and automatic updating (including new malware and even patches to protect against rival botnets).

What differentiates ATP from most botnets and other security threats is the “who”: the ATP exploit tends to be targeted, and brings to bear resources (and patience) indicative of a well-funded actor – most often a nation state. In fact, Greg Hoglund, CEO of HBGary, says ATP is a nice way to not have to say "Chinese state-sponsored threat." Attacks against Google and the U.S. DoD thought to have originated in China would seem to support this definition.

Michael K. Daly of Raytheon, speaking at LISA’09, defines APT more broadly, as increasingly sophisticated cyber attacks by hostile organizations with the goal of:

1. Gaining access to defense, financial and other targeted information from governments, corporations and individuals.
2. Maintaining a foothold in these environments to enable future use and control.
3. Modifying data to disrupt performance in their targets.

But Eddie Schwartz, chief security officer at NetWitness, disagrees that modifying data to disrupt their targets is a universal ATP trait: "A real APT never really damages anything. They tweak a log file here and there ... They are stealing stuff, but you still have your copy. You never see them taint it," he says.

There is no question as to the level of sophistication involved, nor of the value of the assets under siege. Raytheon presents a hypothetical but representative scenario in the diagram below, showing multiple stages, multiple teams, extraordinary stealth and patience, and the exfiltration of well-protected and valuable information assets:

Stage 0 in the diagram above is the "Infection" that gains an initial foothold. How do these infections occur? Damballa points out that APTs can breach target organizations through a wide variety of vectors -- even in the presence of properly designed and maintained defense-in-depth strategies, as shown in the diagram below:

Well-funded APT perpetrators also have the means to compromise insider threats as well as the external threats shown above. Additional "insider threat" and "trusted connection" vectors are shown below:

Advanced persistent threats are in the news these days, and many security vendors are going to great pains to explain how their product (or more likely, the next greatest release of their product) is the ideal solution. But most experts agree that the organizations perpetrating APTs are well-funded, determined, and willing to take as long as necessary to preserve their covert activities. Is it likely that such unique security threats can be adequately addressed by the same technology that was originally developed to solve a different problem? Stay tuned for emerging start-ups such as Cyphort that bring a radically different approach to detecting and remediating advanced persistent threats.

Access Controls, Then and Now

For the past two years I've been telling anyone who will listen that ineffective IT access controls represent an ongoing security vulnerability as well as a compliance liability for many regulated firms. The Ponemon Institute has published a survey that not only confirms what I've been saying, but shows that it's getting worse. What a surprise.

Here's how Ponemon summarizes the problem:

When employees, temporary employees, contractors and partners have inappropriate access to information resources -- that is, access that violates security policies and regulations or that is inappropriate for their current jobs -- companies are subject to serious compliance and business risks.

Fair enough. But many enterprises and security-conscious organizations have a "least privilege" policy to ensure that, as regulations and best practices require, users are provided access to ONLY those resources for which they have a legitimate business need. Doesn't that prevent the inappropriate access referred to above?

Not really. Although least privilege sounds simple enough, in practice it has proven extraordinarily difficult to achieve. This is especially true in dynamic enterprise environments, where activities related to onboarding, offboarding, outsourcing, partnering, and use of contractors threaten to overwhelm whatever business processes exist. These challenges are exacerbated by the coordination required between line-of-business managers, IT staff, HR, security, and compliance staff to manage access controls. In fact, Bruce Schneier, a prominent security guru, states unequivocally that perfect access control just isn't possible. 

Schneier must be on to something. The Ponemon survey, sponsored by Aveksa, found that most relevant metrics for access management are trending down. Here are the top two findings:

• User access rights continue to be poorly managed. Eighty-seven percent of respondents believe that individuals have too much access to information resources that are not pertinent to their job description - up nine percent from the 2008 study.
• Organizations are not able to keep pace with changes to users' job responsibilities and they face serious noncompliance and business risk as a result. Nearly three out of four organizations (72 percent) said they cannot quickly respond to changes in employee access requirements; and more than half (52 percent) reported that they are unable keep pace with the number of access change requests that come in on a regular basis.

What's at risk when access controls are ineffective? Survey respondents' concern was highest for company applications, intellectual property and general business information. Not to mention audit findings.

So what's the primary cause of poor performance in IT access management? A plurality of respondents say "We cannot keep up with our organization's information resources."  This is consistent with Schneier's observation that organizations are simply too chaotic to make it work. So what should be done?

According to the IAM experts, this is where access certification comes in. Here's what Aveksa has to say about access certification:

Good access governance requires the regular review and certification of user entitlements and roles to ensure that access rights to enterprise information assets are appropriate and meet regulatory mandates and guidelines for Sarbanes Oxley, PCI, GLBA, MAR, FERC/NERC, Basel II and HIPAA compliance.  

Many IAM solution providers have integrated modules to help you with your access certification. The problem is, this level of certification -- while important -- involves a review of the rather complicated matrix of staff and roles/entitlement assignments that have overwhelmed organizations in the first place. 

It's not as if organizations don't know they have probable vulnerabilities: the vast majority say it's "likely" that users are over-entitled.

Here's what we can conclude: Organizations suspect that their users have more access than is required, a clear violation of compliance regulations as well as a security risk. And auditors have proven their worst fears, as excessive access rights have remained the top audit finding for years. So we know that organizations are motivated to solve this problem. But despite the availability of comprehensive role-based access control IAM systems, regulated enterprises apparently still do not have the right tools to manage access controls. What they are missing is feedback that quantifies the effectiveness of their access controls.

Current approaches have obviously failed to achieve the desired and necessary level of security and compliance. That's why Cloud Compliance, my prior company, was formed -- to address this and related access audit issues through an innovative SaaS-based capability called Identity and Access Assessment (IdAA). Cloud Compliance provided visibility into not just who is accessing what, but who should access what. And when excessive access rights inevitably occur, Cloud Compliance analytics would help determine the root cause and effective remediation strategies. 

Schubert

My favorite Schubert piano sonata is # 14 in A minor, D.784 (played by Mitsuko Uchida, a piano goddess). It starts by gently probing in the far reaches of our soul, asking ineffable questions that are of the sort one might ponder between dreams. Gradually we are drawn into the A minor universe, rising and falling on the swells of Schubert’s growing tempest. Through the first two movements the dialog progresses as a series of rising storms, sublime wind and currents that dance around themes noble and eternal—separated by interludes of sunlight, not just illumination but light that warms our hearts and enlightens our heads. Urgently and inexorably the melody pushes forward, increasing tension until it can increase no more and then, like a crossbow pulled back one more notch—is it possible?—and then another, and yet another! Finally the third and final movement (allegro vivace) resolves all the built-up tension, thunder and crossbow bolts filling the air with color, pulsing in strict accordance with the inexorable rhythm of the universe, and just as we bring ourselves into confident sync there’s the briefest pause—almost imperceptible—where the force behind the tides of the oceans and orbits of the planets gathers itself for the ecstatic finale. Somehow we’ve journeyed to the far reaches in just under 24 minutes, returning cleansed, fulfilled. I love Schubert's music.

I didn’t really know much about Schubert until a few years ago. And I wasn’t really attracted to classical piano music other than the odd concerto. Too boring compared to instruments that appeal to the ear such as a violin, which when expertly played could bring an audience to tears with a single note. The plaintive tone of an oboe, the rich warmth of the cello, the energy and passion of the brass all strike deeply within whereas the piano seemed to just offer notes. But, inspired by Thomas Mann (Doctor Faustus, chapter VIII) I decided to try again to appreciate the piano—the instrument, unlike all others, for beyond the senses, where what is heard is the noble, intellectual content of the music. Soon I had 10 hours of Beethoven and 9 hours of Schubert piano sonatas on my iPod.

How to deal with so much new music? With Beethoven, it was easy. Of his 32 piano sonatas, 8 or 9 of them became popular enough to have been named (Moonlight, Waldstein, Appassionata, etc.). So I focused on listening to and understanding the named Beethoven piano sonatas as a start.

Schubert was more difficult. I didn’t know where to start, and he didn’t have a list of named sonatas to work with. And so, one Saturday while Jo was in PA, as I was working at home all day, I listened to all 9 hours of Schubert piano sonatas When I heard a theme or phrase I particularly liked I wrote down the sonata that was playing. At the end of the day I had four Schubert piano sonatas to start with.

How do we learn to like pieces of music? For me, the only way is repetition. It takes at least 3 and sometimes 5 or more hearings before I have reached any level of familiarization with any but the simplest tunes. And while we’re at it, what it is about some music that we like and other that we’re not attracted to. In “This Is Your Brain On Music” the author (Daniel J. Levitin) makes the case that one of the attributes of music sophisticated listeners find pleasing is it’s complexity (within the constraints that make it music rather than noise, such as timbre, tempo, etc.). While it’s true that such an theory explains why repeated hearings are required to fully embrace a piece of music, on the whole I found that explanation unsatisfying. The opening bars of Beethoven’s Moonlight Sonata are anything but complex, yet we’re attracted to it nonetheless.

It seems to me there are at least two elements of satisfying music: it’s beauty; and how deeply it touches us, or moves us. And I would think that individuals with different tastes are more likely to agree as to the beauty of a piece of music based on its having a pleasing melody along with well regulated harmony, structure and tempo as per prevailing forms.

But what is it in music that moves us? Personally, for example, I find overwhelming beauty in Bach. I love the St Matthew Passion, the Mass in B minor, Goldberg Variations, Musical Offering, Cello Suites, and others—and listen to them often. But Bach rarely moves me. Same with Mozart; there's beauty, but not much in the way of passion. But Beethoven, Brahms and Schubert do indeed move me with their beautiful music. Why is that? And why is it that someone else might be moved by Back and Mozart, but not Schubert? Dr. Oliver Sacks researches this very topic from a neurological point of view, and shows various portions of the brain “lighting up” more when listening to that music which moves us (in Dr. Sacks’ case, that’s Bach). But I suspect the neurological view is more of the “what” rather than the “why”. Sacks touches on this when he suggests that music is able to reach the oldest, pre-verbal portions of our brain and thus elicit a primal response.

I started playing the four Schubert piano sonatas that somehow made an impression the first time I heard them—sonata #20 in A, sonata #7 in E-flat, and sonata #142 (which, published posthumously, is actually a collection of four impromptus) along with sonata #14 in A minor referred to above. And after listening to them a few times, I found myself drawn to them more and more strongly. I discovered that Schubert’s piano sonatas had the ability to transport me in a way that other pieces could not. I went back and selected other Schubert sonatas to listen to, and my collection of “moving” Schubert piano sonatas began to grow: I’ve now got about 7 or 8 that I listen to on a regular basis.

The following yearI got some Schubert chamber music. Now I have added to my collection of Schubert favorites his “Trout” piano quintet, several string quartets (including “Rosamunde” and “Death and the Maiden”) and the famous Cello Quintet (also published posthumously—he died young—and cited by Wikipedia as deeply sublime, with moments of unique transcendental beauty, and the “high point in the entire chamber repertoire”). In the documentary "Music From the Inside Out", Philadelphia Symphony concertmaster David Kim says the best thing about his career now that he's no longer performing by himself as a traveling violin virtuoso is that he gets to play the Schubert Cello Quintet in a chamber group, which he could never do before.

Schubert’s liturgical music is beautiful, especially his masses; my favorite mass is Schubert’s Mass in E-flat major, although Beethoven's Missa Solemnis, Kodály's Missa Brevis, and of course Bach’s Mass in B minor are favorites as well.

While Schubert in general seems to move me the most, I have found other pieces that do as well: Brahms cello sonata #1, piano quintet in F, string quintet in G, and his sacred choral music; Beethoven’s piano sonata favorites include Moonlight, Waldstein, Appassionata, Tempest and Hammerklavier; I also like his violin sonatas, especially Frühlingsonate and Kreutzer, and his string quarter in F, op. 135. And among the Russians I am especially moved by Tchaikovsky’s Rembrandt Trio and Rachmaninov’s cello sonata and piano concerto #2.

But mostly it's Schubert . He left a fairly large body of work considering the fact he died young (at age 31). He was buried next to Beethoven, whom he greatly admired and who had died the previous year. Many of his manuscripts weren’t found until after he died, and his popularity increased gradually as Robert Schumann and Franz Liszt, among others, transcribed, arranged and promoted his work. On the 100th anniversary of Schubert’s birth in 1897 Vienna celebrated with ten days of Schubert concerts. Imagine that!

Management Thoughts

I get asked about my management and leadership style often. It’s an important question, one that often gets short shrift in silicon valley where engineering execs are hired based on their apparent technical competence (and, when fired, it’s always for other reasons – exercises poor leadership, misses deadlines, or simply doesn’t get the job done).

I’ve had a long-standing interest in questions of motivation, morale, and teamwork: How can a manger inspire a team to a higher level of performance? What motivates people to "be extraordinary" when that is what's required to meet a significant challenge? Many books have been written on that subject and I discuss some influences on my approach to management and leadership below.

The Classics

My influences start with Peter F. Drucker, whose timeless principles still apply in today’s environment. Drucker especially emphasized the value of a company’s human capital and the need to maximize those assets through effective management. He was writing about such core concepts as metrics and management by objectives over fifty years ago. His foundation principles include fairness, and deep respect for everyone in an organization.

Drucker never experienced the tsunami of information today’s managers must cope with. Stephen R. Covey had a lot to say about instilling a discipline – one’s habits – into the practice of management, and how important it is to understand the context within which we work (especially to begin with the end in mind!). But what I like most about Covey is that he brings ethics and principle-centered leadership into the equation. After all, we spend most of our waking hours working; and if we can’t feel good about it other than bringing home a paycheck, then we lose balance in our lives. (There’s a lot of that going around these days.)

There’s a category of books about leadership and achieving high performance from the world of sports, music and other domains which recognizes superior achievement that I think is relevant to business. It’s somewhat ironic that ideas for how to instill principles of unselfishness, making sacrifices to achieve a higher goal, and reaching a mental state (often referred to as “getting into the zone”) in which the highest performance is possible seem to get more attention outside of the business world. But there may be parallels in how, for example, Bill Walsh instilled his “Standard of Performance” on a team widely regarded as one of the worst in sports, so that they became perennial champions. Maybe business leaders would benefit from learning how Michael Tilson Thomas inspires the players in the San Francisco Symphony to perform much better than they ever had under his predecessors, although it’s mostly the same musicians as before. And when Bill Russell describes how he got in a zone and outperformed his competitors to the tune of 11 NBA Championships, aren’t there potentially lessons to be learned from that?

Jim Collins and Level 5 Leadership

Jim Collins is another significant influence. I got a lot out of BHAG (big hairy audacious goals) and "try a lot of stuff and keep what works" in Built to Last; but what inspires me the most is his research on characteristics of an effective leader, summarized in this article in Harvard Business Review, Level 5 Leadership: The Triumph of Humility and Fierce Resolve. Collins identifies a paradoxical combination of personal attributes of the most successful executives: a deep personal humility (giving credit to others for successes, having a calm demeanor); and an intense personal will (unwavering resolve, utterly intolerant of mediocrity). You can think of this balance as the yin and yang of level 5 leadership:

My takeaway is: Get my ego out of the equation, and always set the bar high. The worst thing a leader can do to a team is expect too little. In my opinion, the right formula is this: Start by building a world-class team, communicate the vision, remove roadblocks, maintain high professional standards – and expect great things from the team.

Servant Leadership

Over 40 years ago Robert Greenleaf, founder of the Greenleaf Center for Servant Leadership, published some essays on what he coined servant leadership, a practical philosophy that replaces traditional autocratic leadership with a holistic, ethical approach. Here’s how Greenleaf defined servant leadership:

"The servant-leader is servant first… It begins with the natural feeling that one wants to serve, to serve first. Then conscious choice brings one to aspire to lead. That person is sharply different from one who is leader first, perhaps because of the need to assuage an unusual power drive or to acquire material possessions…The leader-first and the servant-first are two extreme types. Between them there are shadings and blends that are part of the infinite variety of human nature."

Servant leadership applies at the team or departmental level, too. Treating other departments as customers creates the right mind-set for macro-level servant leadership. When this is reciprocated by other departments, great things can happen.

Robert Greenleaf died in 1990, but servant leadership has experienced something of a revival of late, in part due to its association with agile software development methodologies as I’ll explain in greater detail below.

Theory U

C. Otto Scharmer, author of Theory U: Leading From the Future as It Emerges, is a senior lecturer at MIT and a consultant who has helped develop leadership programs for companies such a Google, HP, Daimler, PricewaterhouseCoopers and Fujitsu (as well as non-profits including World Wildlife Fund and African Public Health Leadership Initiative). In his book Scharmer synthesizes and distills ideas from management sciences thought leaders, psychologists, sociologists, captains of industry, and other thoughtful sources on the nature of thinking, social dynamics, motivation, and communication. Scharmer’s work was influenced by, among others, Rudolf Steiner (whom I have read extensively). Because of the depth and breadth of his work, Scharmer is not a particularly easy read but one is well rewarded for the effort invested.

Scharmer’s main thesis is that we have a “blind spot” in our understanding of leadership and transformational change. Scharmer calls it the “invisible dimension” of leadership, even though it is our source dimension, as shown in the figure below:

Scharmer describes a conversation he had with the CEO of a major financial services company. The CEO said that, after years of organizational learning projects and facilitating corporate change, the probability of success of a major project depends on the inner condition, the inner place from which they operate or the source from which all of their actions originate. How can that be improved?

We know a great deal about what leaders do and how they do it. But we know very little about the inner place, the source from which they operate. In professional sports this is an area that gets a great deal of attention. But not so much in the business world.

Think of this example: If we try to understand how Leonardo da Vinci creates a masterpiece such as the Mona Lisa, we can study it in comparison to all other paintings to see if we can apprehend its most salient qualities (the results, or the “what” of his work). Alternatively, we might hope to observe him at work while he’s painting (the process, or the “how”). But what about when he is staring at a blank canvas, just as he’s beginning to paint. What is the source of his art? There’s no question it’s an inner quality—a source dimension. What can we learn about that?

At its core, leadership is about shaping how individuals working as a group attend to a challenge or problem. This is key: Albert Einstein advises us “No problem can be solved from the same level of consciousness that created it.” Creating an environment wherein teams collectively achieve a higher level of consciousness that can be directed towards the challenge at hand is imperative for groups to reach their highest potential. We need help attaining the higher level of consciousness, as Einstein advises, to solve difficult problems. Theory U is about helping teams achieve a higher level of consciousness, which enables them to co-inspire each other to higher levels of performance.

What is Theory U? It’s a process that ultimately enables a team to co-evolve – grow as a group – to achieve the highest collective performance possible. It begins with how we communicate. Typical business conversations are conducted in “downloading” mode: habits of thought are re-confirmed; people talk nice, but generally stick to a predefined script (I knew you were going to say that). The level of communication needs to progress from downloading to factual, object-focused talking and then to the next level – empathetic listening that’s based on inquiry and deep listening. Ultimately that enables the progression through the U-Process, as depicted below:

When one considers the subtitle of the book – leading from the future as it emerges – it’s easy to imagine it as a new-age inspired manifesto rather than the thoughtful, practical source of leadership wisdom that it is. However, the concepts of co-inspiring, co-creating and co-evolving – when applied in an environment of a smart, well-motivated team – can represent the source of a true business advantage.

My Management Style

In describing my management style I am specifically discussing how I apply many of the principles above to the task of being an engineering executive.

Outside of the technical domain, my primary areas of focus are people; and process.

People

The first question of leadership is not what, or how, but who. Jim Collins says about his research into great leaders:

We expected that great leaders would start with vision and strategy. Instead, they attended to people first. They got the right people on the bus, moved the wrong people off, ushered the right people to the right seats – and then figured out where to drive it.

My mandate is to put the best team on the field that I can. Building a world-class team is necessary (but not sufficient) for building a world-class company. In fairness to investors, executives and the rest of the company, we set a high standard and ensure that everyone is performing up to that standard. If they can’t – or won’t – then it’s not a good fit for them and we will replace them with someone who can and will meet our Standard of Performance.

People are my first priority, always. Human capital is the company’s most valuable asset and should be dealt with accordingly – by demanding a high level of contribution, of course, but also by treating them respectfully, fairly and honestly. People won’t be motivated unless they are honored as professionals and human beings worthy of our highest respect. That helps foster the type of company culture that I want to work in, and one that other high performers gravitate towards as well. (One way to foster this is company-wide is to encourage your team to treat other departments as customers.)

But it’s called “work” for a reason, and my attitude towards the people on my team is to ensure we’re performing at as high a level as is reasonably possible. To this end, I focus on the “four C’s”: Communication; Commitment; Competition; and Customers.

Communication: It’s easy to say “communicate more” or “there’s never too much communication” but the truth is more complicated. If one person “communicates” an essential fact to another, but it’s buried on page 27 of a 40-page document and it hasn’t been highlighted or singled out, then effective communication probably hasn’t taken place. On the other hand, if someone is communicating a point of view and the listener is in “downloading” mode, then effective communication probably hasn’t taken place in this case as well either. The only measure of communication is its effectiveness. To communicate well is not just a style issue, it requires thought and effort. We always try to establish processes that support required communications, but in dynamic environments more is required than can be anticipated and defined by standard procedures.

Good communication is hard. Since we mostly work on teams and each team member has access to different information sources, one needs to consider context and other perspectives to determine what information should be communicated. The mode of communication is important as well: much if not most technical information should be communicated in writing, and email is the typical mode (although wikis, blogs and forum discussions may be more effective at times). Sometimes a phone call or face-to-face interactive discussion is the better option, especially if there’s unexpected information, a sensitive topic to discuss, or immediate feedback is required.

Many processes are established to further effective communications, including: status meetings, daily Scrum meetings, distribution lists for key events and updates, product backlog updates and transparent access to development status artifacts, use of and notification rules for tracking databases, performance reviews, and so forth.

A quick word about performance reviews: I’m not a big fan, but they’re mostly necessary – especially when they are tied to annual salary increases.  The main problem I have with reviews is the notion of providing feedback on an annual basis; that’s not enough of that kind of communication, and it comes too late. Instead of feedback I prefer to focus on feedforward: It’s more valuable to an employee to be told in advance what is required -- and in near real time how well the objectives are being met -- than to wait until year-end to find out what should have been done.

Commitment: In general, we make two types of commitments to each other: informal, routine commitments (such as “I’ll send you that report tomorrow”); and more critical, “Capital C” Commitments (such as “this will be released on July 21”). The distinction is important: As we work together, we rely on each other for various tasks, reports, deliverables, pieces of information or other workplace artifacts. Lowercase c commitments are subject to the usual unpredictable events and stresses of the work environment: Of course we intend to meet these commitments, and in most cases we do. But it’s not the end of the world if an unanticipated complication or higher priority interrupt inhibits our ability to meet that commitment (although we should always communicate a change in status if there’s the expectation of a deliverable).

“Capital C” Commitments are different. When such Commitments are made, it must also be made clear (to both parties) that this is critically important. There is a legitimate and urgent need to the business for such Commitments. When unanticipated complications arise or we get interrupted, we are expected to overcome the impact of such inevitable instances of Murphy’s Law. Failure is not an option. We make personal sacrifices, we call upon additional resources if possible (including our own reserves), we work nights and weekends, we do whatever it takes to meet such Commitments. We are measured by our ability to meet and honor such Commitments.

There’s another category of “Capital C” Commitment, and that is in honoring our understandings. In this case, the Commitment is not related to an event or tangible deliverable, but to how we work together, or how we treat each other. For example, I may Commit to directors or managers who report to me that I will not “go around” them in communicating with their staff except under urgent circumstances when they are not available, and if I do I will always give them a heads up. Many such examples could be made; those Commitments need to be taken seriously.

Competition: It’s a tough world, and we have competition: competition for customers; competition for limited venture capital; competition for the best and brightest people; and competition internally for budget dollars, support resources, and so forth. We don’t often deal with competition directly, but it’s a motivating force for much of what we do. Competition is behind why we have a limited budget, why we have a tough deadline, why we add a customer-requested feature late in the release cycle, and why we work so hard.

Competition presents a unique challenge to the development organization: achieving defensible product differentiation. What gets much of our attention in building products and services is the set of prioritized requirements -- referred to as the product backlog in many agile methodologies -- that represents the combined wisdom of customers, sales & marketing, industry analysts and thought leaders, and relevant standards. But great products typically boast of innovative concepts, breakthrough technology, or some form of engineering-conceived special sauce. Engineering needs to fully understand the competitive landscape from a product technology point of view, and should consistently challenge itself to innovate in a way that achieves a measurable and relevant benefit to the customer. That’s a tough standard, but it’s what we like about competition – it brings out the best in us.

Customers: Peter Drucker stated that the purpose of business is to create customers. I had to think about that for a while, but it’s a perfect way of expressing a company’s fundamental objective. And although we expect to do so with great products, positioning, sales, support, operations and financing, there’s one other thing that’s almost always required: good customer references. But why aim so low? The objective, in my opinion, should always be to have great customer references – outstanding references, rave references -- from customers whose trust and respect you’ve earned to the degree that they will go out of their way to tell others about it, emphatically and with conviction. The kind of reference that if a prospect hears, they will almost certainly become a customer. But such a reference can't be asked for, it must be earned.

Ironically, the customers from whom I've received the highest praise in the past were those where something went wrong, sometimes really, embarrassingly wrong. But that’s not what they seemed to remember: What they wanted to talk about – for years, in some cases – was the amount and level of support the customer got when things did go wrong. What we heard from these customers was that they expect things to go wrong from time to time, sometimes seriously so. What they don’t expect is the vendor to pull out all the stops in order to fix whatever’s wrong – in some cases, whether it was our product or not.

Communication; Commitment; Competition; and Customers. If it sounds like a formula, it’s not. People, especially technical professionals, are complex, multi-dimensional and sometimes capable of extraordinary feats. Software development is a group endeavor and therefore a social process. Smart people thrive in an environment where they are surrounded by other smart people with a common goal, where their contributions are honored, where they’re treated with respect, and where they can have fun while they’re accomplishing great things.

Process

Processes should be defined that expedite team efforts rather than getting in the way. There are many aspects to this, but here I focus on agile software development methodologies.

Agile is the term used to refer to a class of development methodologies that are incremental and iterative. Scrum is by far the most popular agile method; others include XP (eXtreme Programming); Unified Process (UP, Agile UP, or Rational UP); Evo; Feature-Driven Development (FDD); Test-Driven Development (TDD); Dynamic Systems Development Method (DSDM); Crystal; and others. There is no “best” approach other than what works for a given team; in many cases, a hybrid is used (for example, Scrum + XP is widely used).

I have used Scrum in multiple settings, and have been impressed by it’s positive impact – especially where requirements are uncertain. For larger-scale developments I would be tempted to blend in aspects of Agile UP since it incorporates more SDLC artifacts into the process. In either case I think it’s important to incorporate test-driven development (TDD) concepts into the agile process.

Why do organizations adopt agile development methods? Generally, it’s for improved visibility to the project status from stakeholder, better adaptability in a dynamic environment, greater business value sooner in the development process, and overall risk reduction according to VersionOne as shown in the chart below:

While these are great reasons, I would say the biggest benefit is that engineers love it compared to traditional approaches, primarily because they are in much greater control over what they do and how they do it. And that’s a Good Thing: There’s no question in my mind that the best outcomes result from a team of well-motivated engineers who feel in control over their environment. But another key reason engineers love agile is it enables them to be heroes in customers eyes, since customer prize responsiveness more than anything. And agile allows development teams to be much more responsive to customer needs.

With Scrum, developers are fundamentally empowered to do their jobs in a team that collaborates together and makes its own technical decisions. According to Ken Schwaber, one of the Scrum founders and author of Agile Software Development With Scrum in 2002, Scrum teams are “self-managing, self-organizing, and cross-functional” and therefore control their own destiny.

But isn’t that like anarchy? Not at all. The Team (Capital T for a Scrum Team) is self-directed, but works from a prioritized Product Backlog that represents the interests of all stakeholders, including executive management. Scrum calls for three roles that work collaboratively to ensure alignment with business objectives and transparency to all stakeholders: Product Owner; ScrumMaster; and Team (typically 6-9 people, but can vary). The Product Owner represents customers’ requirements as well as requirements of other business stakeholders, and determines what are the highest priority features for each sprint (called the Sprint Backlog). On any given sprint it’s the Product Owner’s responsibility to ensure that the team is working on the highest priority items at that time. This is sometimes referred to as “just in time” requirements. And because requirements are dealt with on a just-in-time basis, feature creep and requirements uncertainty are minimized.

The agile project manager for Scrum is referred to as the ScrumMaster. He or she is responsible for ensuring that the Scrum process is followed, and for removing roadblocks as identified by Scrum Team members. Because the nature of agile project management is one of facilitation rather than top-down control, the ideal characteristics of an agile project management role such as ScrumMaster are those of a servant leader. Servant leadership, described as an executive model for large companies in the latter decades of the previous century, has now emerged as the preferred management style for agile software development.

The Scrum process involves one or more time-boxed sprints to a release (typical sprints are 4 weeks or 30 days). Ideally, each sprint should result in potentially releasable code – functionally complete (for the backlog items included in the sprint), refactored if necessary, tested, and documented. Within each sprint, there are daily scrum meetings (sometimes referred to as “stand-ups” – meetings so short and focused that they can be done standing up). At the end of a sprint, a review meeting takes place which usually includes a demo to the Product Owner and any interested stakeholders: One of the tenets of the Agile Manifesto is “working software is the primary measure of progress”.

The following diagram from TargetProcess shows the overall flow:

A variety of Scrum-specific project management tools exist, including from companies such as VersionOne, Atlassian, and Agilebuddy. These tools generally include dashboards for “at a glance” project status as well as burndown charts, defect tracking, velocity trends, analytics and a variety of reports.

Scrum – or for that matter, any agile methodology – can’t be plugged in to an organizational environment as a cookbook formula. The amount of what Craig Larman defines as "ceremony" in Agile and Iterative Development -- what we typically think of as structure (development phases), deliverables (artifacts of the development process), and process (workflow and authorizations) -- are unique to each development, based on dozens of factors. This requires wisdom and judgment, and, like the software being developed, should evolve incrementally and iteratively in every organization.

Summary

There is no single “best” management style; what works for any executive is a function of his or her values, character, and inner condition. In this post I have outlined some of the practices and leadership themes that have worked for me in over 20 years of executive experience.