Bleeding Edge

Leveraging Security Metrics To Protect Your Network

2011-12-21T12:51:00.000-08:00

Maybe we should just give up trying to maintain secureenterprise networks; it’s just too hard. Fully 71% of respondents admitted thattheirnetworks are exposed to external threats due to misconfiguration issuespresent in their security device infrastructure. Verizon reports that 79%of organizations fail to maintain their PCI compliance from their prioryear’s assessment to the next year’s Initial Report on Compliance. More than 50percent had no idea howmany of their organizations’ internal hosts were actually exposed to theInternet.

We know that even in this era of constrained budgets,enterprises are spending more and more on network security—and yet 75% ofnetwork and security pros agree that the advantageis still on the side of the attacker. Verizon reposts that security“erosion” over the course of the year between PCI audits is the case with thevast majority of enterprises, despite the fact that we know there’s a correlationbetween data breaches and lack of PCI compliance.

Maybe it’s time to re-evaluate our priorities. As Dr.Mike points out, there’s a general consensus that much can be gained byfocusing on the basics—the core controls. If you’re covering 90% of the corecontrols, security pros agree it’s better to put effort into getting to 100%rather than expanding the number of controls.

But if you’re focused on the core controls, how do you knowwhat percentage level you’re at, and where the areas of exposure are? That’swhere security metrics comes in.

In this case, we’re referring to actionable securitymetrics—metrics that provide proactivesecurity intelligence. Many metrics are available to security pros: numberof patches; number of vulnerabilities; and the number of firewall and routerconfig changes are good examples of typical metrics. But most of these datapoints are without context, or simply serve as busyness measures. They don’tcharacterize risk in a meaningful way, nor do they point towards a specificarea that needs attention.

Andrew Jaquith,in his book SecurityMetrics: Replacing Fear, Uncertainty and Doubt, describes the value of securitymetrics by contrasting to other business disciplines. For example, freightcompanies know their freight cost per mile and loading factors-as well as thoseof their competitors. Management can therefore set meaningful objectives andmeasure themselves against comparable companies. Choosing to be above, on, orbelow an industry average is a question of strategy as well as operationalefficiency. For example, a freight company may be willing to have a lower loadfactor than its peers if that's the tradeoff required to offer faster deliverytimes (for which it presumably charges a premium).

Similarly, warehousing firms measure and compare theircost/square foot and inventory turns, and e-commerce companies measure theirwebsite conversion rates. And of course financial metrics have beenstandardized and reported on for years. Companies can therefore comparerelevant metrics to those of their peers in order to better evaluate theirinternal performance.

Could such a use of metrics apply to security? Yes, but onlyif consistently generated within the context of a security framework.

The three pillars of security are visualize, comply andprotect. If we build a framework on those pillars we’ll be able to generatemeaningful security metrics.

Visualize: There is wisdom in Requirement 1 of thePCI DSS, in the section entitled “Build and Maintain a Secure Network”: therequirement is to create a network diagram, and keep it current. Why? You can’tsecure what you can’t see. And yet, according to Verizon Requirement1 has the second-highest erosion factor out of the nine requirements notspecific to planning and checking. When security pros can visualize the networktopology—including groups that clearly identify zones (such as DMZ) anduntrusted sources—they become much more effective in creating effectivesegmentation strategies and policies, and maintaining their compliance.

Comply: Compliance refers to PCI, FINRA, FFIEC, SOXand other regulatory frameworks, of course, but also internal policies, andbest practices from sources such as SANS’ 20 Critical Security Controls, Version 3.0. However, complyingwith regulatory and internal policies in most cases is open loop; we performsecurity measures in an effort to comply, but other than regulatory auditswe’re mostly in the dark as to how effective our security controls are. What weneed to do is get from open loop security frameworks to closed loop withfeedback controls that allow us to make continuous adjustments in the presenceof security erosion, as shown in the diagram below:

Protect: The fundamental security question is whetherthe network is protected. How can we know what’s working, and where additionalfocus is required? By developing a security framework that provides security metrics—feedbackcontrols, from which effective remediation strategies to security erosion canbe devised. Security metrics enable enterprise to answer questions such as:

What is my overall level of risk, and how doesit compare to yesterday, last week, last month and last year?
How easily can attackers get in?
How big is my attack surface?
How much of my infrastructure is undocumented?
Are investments and actions paying off?
Where do we need to improve?
Are we ready for our next audit?

Note that the questions above relate to actual networksecurity, unlike, say, how many hosts were patched in the last month (busynessmeasure) or how many vulnerabilities are being scanned for (no context).

Are these good security metrics? Let's look at Andrew Jacquith'sdefinitionof a good metric:

consistently measured, without subjective criteria;
cheap to gather, preferably in an automated way;
expressed as a cardinal number or percentage, not with qualitative labels such as high, medium and low;
expressed using at least one unit of measure, such as "number of hosts directly exposed"; and
contextually specific—relevant enough to decision-makers so that they can take action.

The security metrics provided in RedSeal 5 satisfy all ofJacquith’s criteria for good metrics, enabling RedSeal’s customers tocontinuously monitor their network through a closed loop process and thereforeaddress problem areas—and in doing so protect their organization’s hosts and other sensitive assets.

Mind The Gap

2011-05-01T11:15:00.000-07:00

Often, it's what we don't know that gets us in the most trouble. Take for example the CISO who protects the enterprises cyber assets by firewalls and router ACLs; his or her mental image might look like this:

Why is it, then, that a potential attackers see this as the level of protection?

The CISO has performed all the best practices regarding firewalls and access controls, but attackers invariably find gaps in security.

Bruce Schneier, the internationally recognized security guru, points out

In cyberspace, the balance of power is on the side of the attacker. Attacking a network is much easier than defending a network.

Why? The answer is fairly simple: Enterprise networks have become too complex to analyze their access controls using traditional manual processes. Yes, firewall rules can be reviewed and router ACLs can be examined as best practices dictate. But the actual security is determined by the interaction between dozens of such devices, each with hundreds if not thousands of rules. Even if any given device is providing the intended controls, the total affect may be like the closed gate in the image above: gaps in access controls may exist, and attackers can find them.

Verizon's 2011 Data Breach Investigations Report (DBIR) was recently released; two of its key findings were eye openers:

83% of victims were targets of opportunity;
92% of attacks were not highly difficult.

In other words, eliminating the most visible gaps in security may be enough to discourage cyber attackers who function much like car thieves that walk through parking lots looking checking for unlocked cars. Note that in this example the best way to foil the car thieves is not necessarily to have the best anti-theft technology around, but simply to present more of a challenge to intruders than those who don't lock their cars.

How can this basic level of security be provided, to at least discourage being a target of opportunity? If the complexity of networks defies our ability to manually find access holes, what can be done?

The only possible answer is to employ security analytics that thoroughly examine the interaction between firewalls, routers, load balancers and switches to determine an organization's security posture. RedSeal System's Security Posture Management systems is one of the few solutions available that provides a systematic, automated approach to continuous monitoring of access controls, thereby ensuring that security policies have been effectively implemented and maintained.

Lost in Translation

2011-03-19T15:41:00.000-07:00

In the movie Lost in Translation, Bill Murray's character is shooting a commercial in Japan. The director, who has a clear vision of what he wants, is working through a translator to get his point across. The best the translator can do is say "with intensity". The actual message gets lost in translation.

It's easy to find examples of translations into English gone awry:

Belgrade elevator: To move the cabin push button for wishing floor. If the cabin should enter more persons, each one should press a number for wishing floor. Driving is then going alphabetically by national order.

Rome doctor's office: Specialist in women and other diseases.

Copenhagen airline office: We take your bags and send them in all directions.

Acapulco: The manager has personally passed all water served here.

In enterprise-scale networks, there are usually separate security and network operations departments. The security staff set policies, and rely on the network operations staff to translate these policies into secure network configurations. How reliable is that?

The company I work for, RedSeal Systems, eliminates errors translating between policy and implementation. In addition, it allows the security staff to continuously monitor that the policies have been accurately implemented -- an essential step, given the many configuration changes required in a dynamic enterprise environment. Not only can our solution ensure that nothing gets "lost in translation", we can verify that the security policies continuously remain in force.

"Night Dragon" Latest Reported Advanced Persistent Threat

2011-02-11T11:12:00.000-08:00

Advanced persistent threats, when detected, are rarely publicly reported. Government agencies and enterprises that may have had sensitive data exfiltrated are reluctant to admit it, even more since they are unlikely to know precisely what assets were stolen. That's what makes McAfee's announcement of the so-called Night Dragon exploit noteworthy.

It's been a year since McAfee aired details of Operation Aurora, an advanced persistent threat (APT) that targeted at least 30 companies and organizations -- notably including Google, who publicly linked the exploit to China.

George Kurtz, CTO at McAfee, writes in his blog:

Starting in November 2009, covert cyberattacks were launched against several global oil, energy, and petrochemical companies. The attackers targeted proprietary operations and project-financing information on oil and gas field bids and operations. This information is highly sensitive and can make or break multibillion dollar deals in this extremely competitive industry.
McAfee has identified the tools, techniques, and network activities used in these attacks, which continue on to this day. These attacks have involved an elaborate mix of hacking techniques including social engineering, spear-phishing, Windows exploits, Active Directory compromises, and the use of remote administration tools (RATs).

McAfee provided the graphic below to outline the stages of the attack:

The data accessed by the attackers included operational oil and gas field production systems, financial documents related to field exploration and bidding, and data from SCADA systems.

No one knows how many additional exploits are silently underway, exfiltrating sensitive data, intellectual property and state secrets. What's clear is that the current generation of tools to detect and defend against such attacks are inadequate for preventing such breeches.

Controlling Excessive Entitlements

2011-02-08T11:21:00.000-08:00

Deloitte, in their 2010 Financial Services Global Security Study, reports that excessive entitlements, also known as excessive access rights, was the top audit finding over the past year -- for the third year in a row. It's not an isolated issue: according to Deloitte, excessive entitlements was the top audit finding in retail and commercial banking, insurance, investment banking, and globally across all financial service segments.

Since all major regulatory frameworks, including SOX, PCI DSS, GLBA, NERC and HIPAA, require entitlement controls, many thousands of companies globally are obligated to prevent excessive entitlements and yet, according to the Deloitte survey, have failed to effectively do so.

IDC states that up to 60% of entitlements on most systems are expired and therefore dormant. It's no wonder that auditors can readily uncover excessive entitlements.

Contrast that with entitlements managed by online billing systems, where typically 0% of entitlements are dormant. What's the difference? Why are billing systems able to manage their entitlements effectively, while enterprise IT departments cannot?

The answer? Money.

Billing systems turn entitlements on or off based on payment activity. If an end user stops paying for any reason, the billing system notifies the client company and the associated product or service is no longer made available. If it were not so, the company would lose money by providing products or services for which there is no associated revenue -- in other words, operating at a loss. Because they have a financial incentive to get it right, these companies manage entitlements effectively.

Now consider financial services enterprises. When users are transfered from one department to another, or are assigned new roles in the company, they often retain their legacy entitlements through a transition period for support and training purposes. It's safer to keep these entitlements in case questions come up regarding the prior role. But no real incentive exists for end users to later relinquish their now excessive entitlements, and these entitlements often fall through the cracks of IT and compliance tracking systems. An enterprise may spend hundreds of thousands if not millions of dollars on entitlement management systems. But with up to 60% of accounts in the dormant state, the challenge is simply too great without having line-of-business managers and IT staff spend an unreasonable amount of time trying to stay on top of the issue. As a result, most enterprises have found that effectively managing entitlements and access controls is simply not possible.

Financial incentives work, as demonstrated by online billing systems. So why not try that approach in large enterprises? Considering the risk to the business from failed audits, it's time to think outside of the box. So here's an idea:

What if every user had a payroll deduction for every entitlement that is unused for a certain period, let's say 60 days. The "fine" amount goes into a reserved account, and is refunded once the entitlement is relinquished. This establishes a gentle but real incentive for end users -- not IT, not the compliance group, and certainly not HR -- to manage entitlements. By putting the issue into the hands of the only people who know whether their entitlements are required or not to perform their job functions, and underlining it with a mechanism to ensure visibility and remediation, the problem of excessive entitlements could be solved once and for all.

Advanced Persistent Threats

2011-02-07T13:14:00.000-08:00

Security trends tend to focus on technology: terminology such as malware (on the exploit side) and data leakage protection (on the security solutions side) describe the issue in terms of their most salient technical characteristics. Botnets, drive-by downloads, and Trojan horses add further color to the technical aspects of key security threats.

The “who” behind these security threats is generally thought to be less interesting. Yes, we think we know that certain botnets are controlled by the Russian mafia, and certain exploits tend to be perpetrated by insiders. But it’s the technology behind these threats that we in the high-tech security business use to identify them and their remediation.

Advanced Persistent Threats, recently made trendy by security vendors’ marketing departments, seem fundamentally different. If you look to technical descriptions of advanced, persistent threats (APTs) you will have trouble distinguishing them from botnets. FireEye, for example, describes various command and control systems that bots and APT have in common. Shared characteristics between botnets and APT include stealth, polymorphism (continuously altering malware as it goes from host to host), and automatic updating (including new malware and even patches to protect against rival botnets).

What differentiates ATP from most botnets and other security threats is the “who”: the ATP exploit tends to be targeted, and brings to bear resources (and patience) indicative of a well-funded actor – most often a nation state. In fact, Greg Hoglund, CEO of HBGary, says ATP is a nice way to not have to say "Chinese state-sponsored threat." Attacks against Google and the U.S. DoD thought to have originated in China would seem to support this definition.

Michael K. Daly of Raytheon, speaking at LISA’09, defines APT more broadly, as increasingly sophisticated cyber attacks by hostile organizations with the goal of:

Gaining access to defense, financial and other targeted information from governments, corporations and individuals.
Maintaining a foothold in these environments to enable future use and control.
Modifying data to disrupt performance in their targets.

But Eddie Schwartz, chief security officer at NetWitness, disagrees that modifying data to disrupt their targets is a universal ATP trait: "A real APT never really damages anything. They tweak a log file here and there ... They are stealing stuff, but you still have your copy. You never see them taint it," he says.

There is no question as to the level of sophistication involved, nor of the value of the assets under siege. Raytheon presents a hypothetical but representative scenario in the diagram below, showing multiple stages, multiple teams, extraordinary stealth and patience, and the exfiltration of well-protected and valuable information assets:

Stage 0 in the diagram above is the "Infection" that gains an initial foothold. How do these infections occur? Damballa points out that APTs can breach target organizations through a wide variety of vectors -- even in the presence of properly designed and maintained defense-in-depth strategies, as shown in the diagram below:

Well-funded APT perpetrators also have the means to compromise insider threats as well as the external threats shown above. Additional "insider threat" and "trusted connection" vectors are shown below:

Advanced persistent threats are in the news these days, and many security vendors are going to great pains to explain how their product (or more likely, the next greatest release of their product) is the ideal solution. But most experts agree that the organizations perpetrating APTs are well-funded, determined, and willing to take as long as necessary to preserve their covert activities. Is it likely that such unique security threats can be adequately addressed by the same technology that was originally developed to solve a different problem? Stay tuned for emerging start-ups such as Cyphort that bring a radically different approach to detecting and remediating advanced persistent threats.

Access Controls, Then and Now

2011-01-27T08:16:00.000-08:00

For the past two years I've been telling anyone who will listen that ineffective IT access controls represent an ongoing security vulnerability as well as a compliance liability for many regulated firms. The Ponemon Institute has published a survey that not only confirms what I've been saying, but shows that it's getting worse. What a surprise.

Here's how Ponemon summarizes the problem:

When employees, temporary employees, contractors and partners have inappropriate access to information resources -- that is, access that violates security policies and regulations or that is inappropriate for their current jobs -- companies are subject to serious compliance and business risks.

Fair enough. But many enterprises and security-conscious organizations have a "least privilege" policy to ensure that, as regulations and best practices require, users are provided access to ONLY those resources for which they have a legitimate business need. Doesn't that prevent the inappropriate access referred to above?

Not really. Although least privilege sounds simple enough, in practice it has proven extraordinarily difficult to achieve. This is especially true in dynamic enterprise environments, where activities related to onboarding, offboarding, outsourcing, partnering, and use of contractors threaten to overwhelm whatever business processes exist. These challenges are exacerbated by the coordination required between line-of-business managers, IT staff, HR, security, and compliance staff to manage access controls. In fact, Bruce Schneier, a prominent security guru, states unequivocally that perfect access control just isn't possible.

Schneier must be on to something. The Ponemon survey, sponsored by Aveksa, found that most relevant metrics for access management are trending down. Here are the top two findings:

User access rights continue to be poorly managed. Eighty-seven percent of respondents believe that individuals have too much access to information resources that are not pertinent to their job description - up nine percent from the 2008 study.
Organizations are not able to keep pace with changes to users' job responsibilities and they face serious noncompliance and business risk as a result. Nearly three out of four organizations (72 percent) said they cannot quickly respond to changes in employee access requirements; and more than half (52 percent) reported that they are unable keep pace with the number of access change requests that come in on a regular basis.

What's at risk when access controls are ineffective? Survey respondents' concern was highest for company applications, intellectual property and general business information. Not to mention audit findings.

So what's the primary cause of poor performance in IT access management? A plurality of respondents say "We cannot keep up with our organization's information resources." This is consistent with Schneier's observation that organizations are simply too chaotic to make it work. So what should be done?

According to the IAM experts, this is where access certification comes in. Here's what Aveksa has to say about access certification:

Good access governance requires the regular review and certification of user entitlements and roles to ensure that access rights to enterprise information assets are appropriate and meet regulatory mandates and guidelines for Sarbanes Oxley, PCI, GLBA, MAR, FERC/NERC, Basel II and HIPAA compliance.

Many IAM solution providers have integrated modules to help you with your access certification. The problem is, this level of certification -- while important -- involves a review of the rather complicated matrix of staff and roles/entitlement assignments that have overwhelmed organizations in the first place.

It's not as if organizations don't know they have probable vulnerabilities: the vast majority say it's "likely" that users are over-entitled.

Here's what we can conclude: Organizations suspect that their users have more access than is required, a clear violation of compliance regulations as well as a security risk. And auditors have proven their worst fears, as excessive access rights have remained the top audit finding for years. So we know that organizations are motivated to solve this problem. But despite the availability of comprehensive role-based access control IAM systems, regulated enterprises apparently still do not have the right tools to manage access controls. What they are missing is feedback that quantifies the effectiveness of their access controls.

Current approaches have obviously failed to achieve the desired and necessary level of security and compliance. That's why Cloud Compliance, my prior company, was formed -- to address this and related access audit issues through an innovative SaaS-based capability called Identity and Access Assessment (IdAA). Cloud Compliance provided visibility into not just who is accessing what, but who should access what. And when excessive access rights inevitably occur, Cloud Compliance analytics would help determine the root cause and effective remediation strategies.

Schubert

2011-01-01T12:39:00.000-08:00

My favorite Schubert piano sonata is # 14 in A minor, D.784 (played by Mitsuko Uchida, a piano goddess). It starts by gently probing in the far reaches of our soul, asking ineffable questions that are of the sort one might ponder between dreams. Gradually we are drawn into the A minor universe, rising and falling on the swells of Schubert’s growing tempest. Through the first two movements the dialog progresses as a series of rising storms, sublime wind and currents that dance around themes noble and eternal—separated by interludes of sunlight, not just illumination but light that warms our hearts and enlightens our heads. Urgently and inexorably the melody pushes forward, increasing tension until it can increase no more and then, like a crossbow pulled back one more notch—is it possible?—and then another, and yet another! Finally the third and final movement (allegro vivace) resolves all the built-up tension, thunder and crossbow bolts filling the air with color, pulsing in strict accordance with the inexorable rhythm of the universe, and just as we bring ourselves into confident sync there’s the briefest pause—almost imperceptible—where the force behind the tides of the oceans and orbits of the planets gathers itself for the ecstatic finale. Somehow we’ve journeyed to the far reaches in just under 24 minutes, returning cleansed, fulfilled. I love Schubert's music.

I didn’t really know much about Schubert until a few years ago. And I wasn’t really attracted to classical piano music other than the odd concerto. Too boring compared to instruments that appeal to the ear such as a violin, which when expertly played could bring an audience to tears with a single note. The plaintive tone of an oboe, the rich warmth of the cello, the energy and passion of the brass all strike deeply within whereas the piano seemed to just offer notes. But, inspired by Thomas Mann (Doctor Faustus, chapter VIII) I decided to try again to appreciate the piano—the instrument, unlike all others, for beyond the senses, where what is heard is the noble, intellectual content of the music. Soon I had 10 hours of Beethoven and 9 hours of Schubert piano sonatas on my iPod.

How to deal with so much new music? With Beethoven, it was easy. Of his 32 piano sonatas, 8 or 9 of them became popular enough to have been named (Moonlight, Waldstein, Appassionata, etc.). So I focused on listening to and understanding the named Beethoven piano sonatas as a start.

Schubert was more difficult. I didn’t know where to start, and he didn’t have a list of named sonatas to work with. And so, one Saturday while Jo was in PA, as I was working at home all day, I listened to all 9 hours of Schubert piano sonatas When I heard a theme or phrase I particularly liked I wrote down the sonata that was playing. At the end of the day I had four Schubert piano sonatas to start with.

How do we learn to like pieces of music? For me, the only way is repetition. It takes at least 3 and sometimes 5 or more hearings before I have reached any level of familiarization with any but the simplest tunes. And while we’re at it, what it is about some music that we like and other that we’re not attracted to. In “This Is Your Brain On Music” the author (Daniel J. Levitin) makes the case that one of the attributes of music sophisticated listeners find pleasing is it’s complexity (within the constraints that make it music rather than noise, such as timbre, tempo, etc.). While it’s true that such an theory explains why repeated hearings are required to fully embrace a piece of music, on the whole I found that explanation unsatisfying. The opening bars of Beethoven’s Moonlight Sonata are anything but complex, yet we’re attracted to it nonetheless.

It seems to me there are at least two elements of satisfying music: it’s beauty; and how deeply it touches us, or moves us. And I would think that individuals with different tastes are more likely to agree as to the beauty of a piece of music based on its having a pleasing melody along with well regulated harmony, structure and tempo as per prevailing forms.

But what is it in music that moves us? Personally, for example, I find overwhelming beauty in Bach. I love the St Matthew Passion, the Mass in B minor, Goldberg Variations, Musical Offering, Cello Suites, and others—and listen to them often. But Bach rarely moves me. Same with Mozart; there's beauty, but not much in the way of passion. But Beethoven, Brahms and Schubert do indeed move me with their beautiful music. Why is that? And why is it that someone else might be moved by Back and Mozart, but not Schubert? Dr. Oliver Sacks researches this very topic from a neurological point of view, and shows various portions of the brain “lighting up” more when listening to that music which moves us (in Dr. Sacks’ case, that’s Bach). But I suspect the neurological view is more of the “what” rather than the “why”. Sacks touches on this when he suggests that music is able to reach the oldest, pre-verbal portions of our brain and thus elicit a primal response.

I started playing the four Schubert piano sonatas that somehow made an impression the first time I heard them—sonata #20 in A, sonata #7 in E-flat, and sonata #142 (which, published posthumously, is actually a collection of four impromptus) along with sonata #14 in A minor referred to above. And after listening to them a few times, I found myself drawn to them more and more strongly. I discovered that Schubert’s piano sonatas had the ability to transport me in a way that other pieces could not. I went back and selected other Schubert sonatas to listen to, and my collection of “moving” Schubert piano sonatas began to grow: I’ve now got about 7 or 8 that I listen to on a regular basis.

The following yearI got some Schubert chamber music. Now I have added to my collection of Schubert favorites his “Trout” piano quintet, several string quartets (including “Rosamunde” and “Death and the Maiden”) and the famous Cello Quintet (also published posthumously—he died young—and cited by Wikipedia as deeply sublime, with moments of unique transcendental beauty, and the “high point in the entire chamber repertoire”). In the documentary "Music From the Inside Out", Philadelphia Symphony concertmaster David Kim says the best thing about his career now that he's no longer performing by himself as a traveling violin virtuoso is that he gets to play the Schubert Cello Quintet in a chamber group, which he could never do before.

Schubert’s liturgical music is beautiful, especially his masses; my favorite mass is Schubert’s Mass in E-flat major, although Beethoven's Missa Solemnis, Kodály's Missa Brevis, and of course Bach’s Mass in B minor are favorites as well.

While Schubert in general seems to move me the most, I have found other pieces that do as well: Brahms cello sonata #1, piano quintet in F, string quintet in G, and his sacred choral music; Beethoven’s piano sonata favorites include Moonlight, Waldstein, Appassionata, Tempest and Hammerklavier; I also like his violin sonatas, especially Frühlingsonate and Kreutzer, and his string quarter in F, op. 135. And among the Russians I am especially moved by Tchaikovsky’s Rembrandt Trio and Rachmaninov’s cello sonata and piano concerto #2.

But mostly it's Schubert . He left a fairly large body of work considering the fact he died young (at age 31). He was buried next to Beethoven, whom he greatly admired and who had died the previous year. Many of his manuscripts weren’t found until after he died, and his popularity increased gradually as Robert Schumann and Franz Liszt, among others, transcribed, arranged and promoted his work. On the 100th anniversary of Schubert’s birth in 1897 Vienna celebrated with ten days of Schubert concerts. Imagine that!

Optimizing Pricing in the Real World

2010-10-10T13:06:00.000-07:00

Pricing is an area that most CMOs would agree is sometimes black art and sometimes science. Innovations are rare: The airlines developed yield management strategies in the 1990s, where the price of a seat varied based on a number of factors including the amount of time until takeoff. Online markets such a eBay and Priceline.com offer "bid and ask" exchanges but are largely relegated to B2C businesses.

Pricing strategies generally attempt to achieve some balance of maximizing revenues, profits or market share. But it's impossible to know in advance the effect of promotions, bundles, upgrades, coupons, and plan changes -- not to mention customers' willingness to pay. And there are many factors outside of a business's control that may come into play, including macro economic trends, competitive responses, market dynamics and changing consumer preferences. While analytical models are useful in establishing an initial pricing strategy, tools for measuring the results of price tweaks, promotions and new bundles based on real world experiences have been in short supply. Until now.

Online billing and subscriber lifecycle management platforms provide the ability to measure the results of -- and therefore optimize -- pricing. Any new product bundle, service plan or price change can affect customer uptake and revenue per customer. Some such changes will be more beneficial than others. An online billing solution with appropriate reporting and analytics can measure pricing/bundle/promotion combinations against each other to determine which is most effective and should be widely deployed.

Consider this example: A new SaaS offering includes a free, standard edition as well as a paid-for premium edition. Options are also provided for additional storage. Some customers may be offered the premium plan but not the standard edition, with varying free trial periods. Other related products may be offered separately or in bundles. Competitive benchmarks are used to set initial pricing, and a number of solution bundles are offered. In addition, certain plans have associated upgrades that may be offered to achieve maximum revenue from a customer. With this many variables, how can a business choose the optimal pricing strategy?

A comprehensive billing and subscription management solution such as that offered by Aria Systems provides a means to optimize pricing based on actual subscriber behaviors and preferences. Each pricing, bundling and upgrade plan can be compared to others in terms of adoption rates and average revenue per user (ARPU). Those plans that result in the least revenue or adoption rate can be discontinued, while those that work the best in light of the company strategy can be more aggressively promoted. Over time, as market dynamics change, these plans can be tweaked on an ongoing basis to enable the highest possible revenue from the solutions being offered.

Support for a variety of plans is an architectural capability of advanced billing systems, and only Aria Systems' solution is based on such an architecture. With Aria, it's as if a business can operate with a pricing dashboard, rather than flying blind. And the same multi-plan architecture that enables pricing optimization also provides channel support -- an essential component of any subscriber management system in the subscription economy ecosphere. Aria Systems is unique among online billing providers in enabling real-world pricing optimization as well as comprehensive channel support.

Aria Systems, the leading provider of on-demand billing solutions, offers the only monetization platform encompassing the full spectrum of Billing and Subscription Management services.

Why Subscriber Lifecycle Management Is Like SEM - Only Better

2010-09-10T10:50:00.000-07:00

Search Engine Marketing, or SEM, has revolutionized the customer acquisition process. According to Andreas Ramos and Stephanie Cota in their book Search Engine Marketing, SEM -- also known as search engine optimization (SEO) and online marketing -- includes targeted messaging, pay per click or paid search, analytics, multivariate testing, business intelligence, and CRM. These tools enable deep understanding of prospective customer behaviors as they advance through the buying cycle, from awareness of need to research and comparison to purchase. Once the prospect becomes a customer, these tools no longer apply.

For many revenue models, especially for legacy businesses, this approach works fine. But as cloud-based offerings usher in the subscription economy, decision support tools that conclude with a purchase -- presumably involving a one-time shopping cart transaction -- fall well short of providing necessary visibility to subscriber behaviors.

In the subscription economy, the customer enters into a relationship that typically involves ongoing delivery of value in the form of software or services. The revenue model is recurring. Customers pay as they go, usually without an upfront purchase requirement. In the so-called "freemium" model there is no payment required for a basic or enhanced edition, but an upgrade to a premium product or service is offered. In all such cases, the customer enters into a relationship with their provider. The lifecycle of that relationship may include promotions, upgrades, "sidegrades", tiered plans, support, and other revenue enhancement opportunities.

What tools do modern businesses have to manage the subscriber lifecycle? Certainly not SEM or SEO, whose mission ends with an initial purchase.

Emerging online billing systems provide a platform for subscriber lifecycle management, by leveraging billing-related customer touch points such as invoicing, collection, presentation as well as subscription-related notifications including renewals, credit card expiration and so forth. By using analytics and reporting capabilities of such systems, businesses can gain deep insights into customer behaviors and in doing so fine-tune their offerings and pricing -- and thus maximize their subscription revenues. In other words, the same types of decision support tools as provided by SEM solutions for analysis of behaviors prior to a one-time purchase are provided by subscriber lifecycle management platforms for ongoing analysis of subscribers' behaviors, even long after the initial subscription.

Aria Systems is the leading provider of on-demand billing solutions and offers the only Subscriber Lifecycle Management platform encompassing the most comprehensive set of billing and subscription management services.

Billing As A Service

2010-08-23T17:42:00.000-07:00

A new generation of web billing solutions has emerged in support of recurring revenue business models. These solutions are known by several different names including on-demand billing, subscription billing, and billing as a service. Whatever you call them, these solutions address the following market needs:

SaaS vendors that deliver software on a subscription basis rather than as a one-time purchase are in need of a solution to manage billing in light of different service levels, promotions, upgrades and enhancements.
The "freemium" business model used by many smart phones apps needs a comprehensive billing solution that's radically different from the venerable shopping cart.
The unpredictable, real-time need for micropayments in online gaming can't be economically addressed by simply charging the user's credit card for each individual purchase.
Telecommunications has always been billed on a monthly basis; new service provider entrants can benefit by leveraging web billing services rather than sinking millions into yet another back-office system.
Online publishers offer tiers of access to content, and need to support micropayments for individual purchases.
Online retail and franchise- or agent-based financial services businesses, insurance, energy, and utilities are planning or have recently launched subscription-based products or services and will need secure, scalable recurring billing capabilities.

In all of these examples, there is a need for secure and flexible support for user self-registration, activation, and service level management or customers won't be able to scale their business. They also need currency conversions and country-specific tax calculations if they intend to sell internationally. Then there's exception processing: What if the credit card expires half-way through a one-year subscription? What if the user is due a refund for some service anomaly? What if fraud is detected?

In all of the examples cited above, we're talking about how an emerging business is monetized. That gets people's attention, including from analyst firms such as IDC and Saugatuck. According to IDC:

Billing as a service is a natural development in the emerging SaaS (application services) and cloud (infrastructure resource services) business services market. Not only are billing services core to the online operations of any ecommerce site, but they are also of particular importance in the business of customer monetization for SaaS and cloud software providers themselves, as the market shifts from on-premise software licensing models to subscription- or demand-based pricing and billing.

Although there are a number of vendors who provide billing solutions, three start-ups have attracted attention in this emerging field: Zuora; Vindicia; and Aria Systems. Each of these companies has a unique perspective, shaped by their respective backgrounds. Zuora's founder and CEO came out of Salesforce.com, and approaches the billing as a service space with a deep understanding of SaaS and CRM. Vindicia's CEO, CTO and EVP Engineering previously worked at eMusic, which had its own subscription service and because if the nature of their business had to have an especially strong fraud management solution. Aria Systems' founders came out of the telecommunications/ISP space, known for having some of the most comprehensive (and complex) billing systems of any industry.

Ed Sullivan, founder of Aria, describes a common scenario that serves as an example of how recurring billing might be more complicated than it first appears:

For example, let’s say you have a customer referred to you by a commissioned reseller to be billed monthly for a usage based service. In the course of the month, they suffer a service outage and dispute their bill. How would your billing system handle this scenario? Would it be able to offer a service credit, a refund or a dollar amount credit? How would your billing system deal with the reseller’s commissions, and to make it more complicated, what would your system do if the reseller was on a multi-tiered commission level?

An automated subscription billing system can handle the exceptions detailed above because it goes through the reconciliation process before calculating the invoice. A subscription billing service should be built on five pillars: reconciliation, calculation, presentation, collections and remittance. Our system is set up to go through this process automatically every time before an invoice is presented.

The "billing as a service" label sells this category short, since vendors such as Aria Systems go beyond billing to provide a complete subscription management solution. Here's an example, again from Ed Sullivan of Aria:

With access to real-time information on how different customers are using the software, it becomes a fairly simple exercise to create different pricing schemes for different user groups and decide how best to construct pricing. Evidence may show, for example, that for one company, the most profitable and user-accepted pricing plan would combine a monthly charge with a charge per seat. Yet another company might find that pricing based on levels of use (utility-based billing) would be more appropriate. Either way, the ability to see how customers are using software provides pricing flexibility, which in turn leads to much greater revenue potential.

Companies can aim even higher than pricing by thinking bigger. For example, when you offer software as a service via the cloud, it gives software developers and other third parties the opportunity to create market-specific products based on your offering, as well as plug-ins, extensions and even new features of your core product.

Each such development not only increases revenue for your base offering, but also can provide you with a recurring revenue stream via a percentage of every sale the third-party developer makes.

Of the three on-demand billing vendors Aria Systems has emerged as the market leader, supporting over 2.5 million users in over 200 countries at the rate of over $1 billion per year in transactions -- and growing fast. Saugatuck lists the following as Aria's specific strengths in the marketplace:

A functionally complete, robust telco-grade billing and payments solution with exceptional customer and subscriber management and exceptional partner and reseller management capabilities
End-to-end PCI and SAS 70 Level 1 compliance
Fraud management capability developed for the online gaming industry
Support for more than 20 payment gateways and processors, including Chase PaymentTech, Global Collect, PayPal, and CyberSource's Authorize.net

Another capability that has enhanced Aria's market position is support for two-way synchronization with NetSuite and salesforce.com.

Aria's success to date has attracted the attention of leading investors and market analysts. The highly respected venture firm Venrock led the most recent round of financing. Aria was recently named a "Cool Vendor" by Gartner, and was recognized as a "Top 50" startup at TiEcom 2010. Even more impressive is the fact that PayPal, the worldwide leader in online payment services, selected Aria for its subscription management solution, an endorsement that generated a fair bit of buzz -- see here and here.

Aria was the original pioneer of demand billing solutions six years ago, and has steadily grown their business since. Again, quoting from the Saugatuck report:

Aria's differentiation focuses on delivering a robust, proven billing and customer management solution that can monetize Cloud solutions and other recurring revenue or subscription businesses. In delivering the Aria Billing solution, in both Standard and Enterprise editions, Aria claims a six-year record of success, and the ability to manage, transparently, very high volumes of transactions for its customers.

Management Thoughts

2010-07-08T10:54:00.000-07:00

I get asked about my management and leadership style often. It’s an important question, one that often gets short shrift in silicon valley where engineering execs are hired based on their apparent technical competence (and, when fired, it’s always for other reasons – exercises poor leadership, misses deadlines, or simply doesn’t get the job done).

I’ve had a long-standing interest in questions of motivation, morale, and teamwork: How can a manger inspire a team to a higher level of performance? What motivates people to "be extraordinary" when that is what's required to meet a significant challenge? Many books have been written on that subject and I discuss some influences on my approach to management and leadership below.

The Classics

My influences start with Peter F. Drucker, whose timeless principles still apply in today’s environment. Drucker especially emphasized the value of a company’s human capital and the need to maximize those assets through effective management. He was writing about such core concepts as metrics and management by objectives over fifty years ago. His foundation principles include fairness, and deep respect for everyone in an organization.

Drucker never experienced the tsunami of information today’s managers must cope with. Stephen R. Covey had a lot to say about instilling a discipline – one’s habits – into the practice of management, and how important it is to understand the context within which we work (especially to begin with the end in mind!). But what I like most about Covey is that he brings ethics and principle-centered leadership into the equation. After all, we spend most of our waking hours working; and if we can’t feel good about it other than bringing home a paycheck, then we lose balance in our lives. (There’s a lot of that going around these days.)

There’s a category of books about leadership and achieving high performance from the world of sports, music and other domains which recognizes superior achievement that I think is relevant to business. It’s somewhat ironic that ideas for how to instill principles of unselfishness, making sacrifices to achieve a higher goal, and reaching a mental state (often referred to as “getting into the zone”) in which the highest performance is possible seem to get more attention outside of the business world. But there may be parallels in how, for example, Bill Walsh instilled his “Standard of Performance” on a team widely regarded as one of the worst in sports, so that they became perennial champions. Maybe business leaders would benefit from learning how Michael Tilson Thomas inspires the players in the San Francisco Symphony to perform much better than they ever had under his predecessors, although it’s mostly the same musicians as before. And when Bill Russell describes how he got in a zone and outperformed his competitors to the tune of 11 NBA Championships, aren’t there potentially lessons to be learned from that?

Jim Collins and Level 5 Leadership

Jim Collins is another significant influence. I got a lot out of BHAG (big hairy audacious goals) and "try a lot of stuff and keep what works" in Built to Last; but what inspires me the most is his research on characteristics of an effective leader, summarized in this article in Harvard Business Review, Level 5 Leadership: The Triumph of Humility and Fierce Resolve. Collins identifies a paradoxical combination of personal attributes of the most successful executives: a deep personal humility (giving credit to others for successes, having a calm demeanor); and an intense personal will (unwavering resolve, utterly intolerant of mediocrity). You can think of this balance as the yin and yang of level 5 leadership:

My takeaway is: Get my ego out of the equation, and always set the bar high. The worst thing a leader can do to a team is expect too little. In my opinion, the right formula is this: Start by building a world-class team, communicate the vision, remove roadblocks, maintain high professional standards – and expect great things from the team.

Servant Leadership

Over 40 years ago Robert Greenleaf, founder of the Greenleaf Center for Servant Leadership, published some essays on what he coined servant leadership, a practical philosophy that replaces traditional autocratic leadership with a holistic, ethical approach. Here’s how Greenleaf defined servant leadership:

"The servant-leader is servant first… It begins with the natural feeling that one wants to serve, to serve first. Then conscious choice brings one to aspire to lead. That person is sharply different from one who is leader first, perhaps because of the need to assuage an unusual power drive or to acquire material possessions…The leader-first and the servant-first are two extreme types. Between them there are shadings and blends that are part of the infinite variety of human nature."

Servant leadership applies at the team or departmental level, too. Treating other departments as customers creates the right mind-set for macro-level servant leadership. When this is reciprocated by other departments, great things can happen.

Robert Greenleaf died in 1990, but servant leadership has experienced something of a revival of late, in part due to its association with agile software development methodologies as I’ll explain in greater detail below.

Theory U

C. Otto Scharmer, author of Theory U: Leading From the Future as It Emerges, is a senior lecturer at MIT and a consultant who has helped develop leadership programs for companies such a Google, HP, Daimler, PricewaterhouseCoopers and Fujitsu (as well as non-profits including World Wildlife Fund and African Public Health Leadership Initiative). In his book Scharmer synthesizes and distills ideas from management sciences thought leaders, psychologists, sociologists, captains of industry, and other thoughtful sources on the nature of thinking, social dynamics, motivation, and communication. Scharmer’s work was influenced by, among others, Rudolf Steiner (whom I have read extensively). Because of the depth and breadth of his work, Scharmer is not a particularly easy read but one is well rewarded for the effort invested.

Scharmer’s main thesis is that we have a “blind spot” in our understanding of leadership and transformational change. Scharmer calls it the “invisible dimension” of leadership, even though it is our source dimension, as shown in the figure below:

Scharmer describes a conversation he had with the CEO of a major financial services company. The CEO said that, after years of organizational learning projects and facilitating corporate change, the probability of success of a major project depends on the inner condition, the inner place from which they operate or the source from which all of their actions originate. How can that be improved?

We know a great deal about what leaders do and how they do it. But we know very little about the inner place, the source from which they operate. In professional sports this is an area that gets a great deal of attention. But not so much in the business world.

Think of this example: If we try to understand how Leonardo da Vinci creates a masterpiece such as the Mona Lisa, we can study it in comparison to all other paintings to see if we can apprehend its most salient qualities (the results, or the “what” of his work). Alternatively, we might hope to observe him at work while he’s painting (the process, or the “how”). But what about when he is staring at a blank canvas, just as he’s beginning to paint. What is the source of his art? There’s no question it’s an inner quality—a source dimension. What can we learn about that?

At its core, leadership is about shaping how individuals working as a group attend to a challenge or problem. This is key: Albert Einstein advises us “No problem can be solved from the same level of consciousness that created it.” Creating an environment wherein teams collectively achieve a higher level of consciousness that can be directed towards the challenge at hand is imperative for groups to reach their highest potential. We need help attaining the higher level of consciousness, as Einstein advises, to solve difficult problems. Theory U is about helping teams achieve a higher level of consciousness, which enables them to co-inspire each other to higher levels of performance.

What is Theory U? It’s a process that ultimately enables a team to co-evolve – grow as a group – to achieve the highest collective performance possible. It begins with how we communicate. Typical business conversations are conducted in “downloading” mode: habits of thought are re-confirmed; people talk nice, but generally stick to a predefined script (I knew you were going to say that). The level of communication needs to progress from downloading to factual, object-focused talking and then to the next level – empathetic listening that’s based on inquiry and deep listening. Ultimately that enables the progression through the U-Process, as depicted below:

When one considers the subtitle of the book – leading from the future as it emerges – it’s easy to imagine it as a new-age inspired manifesto rather than the thoughtful, practical source of leadership wisdom that it is. However, the concepts of co-inspiring, co-creating and co-evolving – when applied in an environment of a smart, well-motivated team – can represent the source of a true business advantage.

My Management Style

In describing my management style I am specifically discussing how I apply many of the principles above to the task of being an engineering executive.

Outside of the technical domain, my primary areas of focus are people; and process.

People

The first question of leadership is not what, or how, but who. Jim Collins says about his research into great leaders:

We expected that great leaders would start with vision and strategy. Instead, they attended to people first. They got the right people on the bus, moved the wrong people off, ushered the right people to the right seats – and then figured out where to drive it.

My mandate is to put the best team on the field that I can. Building a world-class team is necessary (but not sufficient) for building a world-class company. In fairness to investors, executives and the rest of the company, we set a high standard and ensure that everyone is performing up to that standard. If they can’t – or won’t – then it’s not a good fit for them and we will replace them with someone who can and will meet our Standard of Performance.

People are my first priority, always. Human capital is the company’s most valuable asset and should be dealt with accordingly – by demanding a high level of contribution, of course, but also by treating them respectfully, fairly and honestly. People won’t be motivated unless they are honored as professionals and human beings worthy of our highest respect. That helps foster the type of company culture that I want to work in, and one that other high performers gravitate towards as well. (One way to foster this is company-wide is to encourage your team to treat other departments as customers.)

But it’s called “work” for a reason, and my attitude towards the people on my team is to ensure we’re performing at as high a level as is reasonably possible. To this end, I focus on the “four C’s”: Communication; Commitment; Competition; and Customers.

Communication: It’s easy to say “communicate more” or “there’s never too much communication” but the truth is more complicated. If one person “communicates” an essential fact to another, but it’s buried on page 27 of a 40-page document and it hasn’t been highlighted or singled out, then effective communication probably hasn’t taken place. On the other hand, if someone is communicating a point of view and the listener is in “downloading” mode, then effective communication probably hasn’t taken place in this case as well either. The only measure of communication is its effectiveness. To communicate well is not just a style issue, it requires thought and effort. We always try to establish processes that support required communications, but in dynamic environments more is required than can be anticipated and defined by standard procedures.

Good communication is hard. Since we mostly work on teams and each team member has access to different information sources, one needs to consider context and other perspectives to determine what information should be communicated. The mode of communication is important as well: much if not most technical information should be communicated in writing, and email is the typical mode (although wikis, blogs and forum discussions may be more effective at times). Sometimes a phone call or face-to-face interactive discussion is the better option, especially if there’s unexpected information, a sensitive topic to discuss, or immediate feedback is required.

Many processes are established to further effective communications, including: status meetings, daily Scrum meetings, distribution lists for key events and updates, product backlog updates and transparent access to development status artifacts, use of and notification rules for tracking databases, performance reviews, and so forth.

A quick word about performance reviews: I’m not a big fan, but they’re mostly necessary – especially when they are tied to annual salary increases. The main problem I have with reviews is the notion of providing feedback on an annual basis; that’s not enough of that kind of communication, and it comes too late. Instead of feedback I prefer to focus on feedforward: It’s more valuable to an employee to be told in advance what is required -- and in near real time how well the objectives are being met -- than to wait until year-end to find out what should have been done.

Commitment: In general, we make two types of commitments to each other: informal, routine commitments (such as “I’ll send you that report tomorrow”); and more critical, “Capital C” Commitments (such as “this will be released on July 21”). The distinction is important: As we work together, we rely on each other for various tasks, reports, deliverables, pieces of information or other workplace artifacts. Lowercase c commitments are subject to the usual unpredictable events and stresses of the work environment: Of course we intend to meet these commitments, and in most cases we do. But it’s not the end of the world if an unanticipated complication or higher priority interrupt inhibits our ability to meet that commitment (although we should always communicate a change in status if there’s the expectation of a deliverable).

“Capital C” Commitments are different. When such Commitments are made, it must also be made clear (to both parties) that this is critically important. There is a legitimate and urgent need to the business for such Commitments. When unanticipated complications arise or we get interrupted, we are expected to overcome the impact of such inevitable instances of Murphy’s Law. Failure is not an option. We make personal sacrifices, we call upon additional resources if possible (including our own reserves), we work nights and weekends, we do whatever it takes to meet such Commitments. We are measured by our ability to meet and honor such Commitments.

There’s another category of “Capital C” Commitment, and that is in honoring our understandings. In this case, the Commitment is not related to an event or tangible deliverable, but to how we work together, or how we treat each other. For example, I may Commit to directors or managers who report to me that I will not “go around” them in communicating with their staff except under urgent circumstances when they are not available, and if I do I will always give them a heads up. Many such examples could be made; those Commitments need to be taken seriously.

Competition: It’s a tough world, and we have competition: competition for customers; competition for limited venture capital; competition for the best and brightest people; and competition internally for budget dollars, support resources, and so forth. We don’t often deal with competition directly, but it’s a motivating force for much of what we do. Competition is behind why we have a limited budget, why we have a tough deadline, why we add a customer-requested feature late in the release cycle, and why we work so hard.

Competition presents a unique challenge to the development organization: achieving defensible product differentiation. What gets much of our attention in building products and services is the set of prioritized requirements -- referred to as the product backlog in many agile methodologies -- that represents the combined wisdom of customers, sales & marketing, industry analysts and thought leaders, and relevant standards. But great products typically boast of innovative concepts, breakthrough technology, or some form of engineering-conceived special sauce. Engineering needs to fully understand the competitive landscape from a product technology point of view, and should consistently challenge itself to innovate in a way that achieves a measurable and relevant benefit to the customer. That’s a tough standard, but it’s what we like about competition – it brings out the best in us.

Customers: Peter Drucker stated that the purpose of business is to create customers. I had to think about that for a while, but it’s a perfect way of expressing a company’s fundamental objective. And although we expect to do so with great products, positioning, sales, support, operations and financing, there’s one other thing that’s almost always required: good customer references. But why aim so low? The objective, in my opinion, should always be to have great customer references – outstanding references, rave references -- from customers whose trust and respect you’ve earned to the degree that they will go out of their way to tell others about it, emphatically and with conviction. The kind of reference that if a prospect hears, they will almost certainly become a customer. But such a reference can't be asked for, it must be earned.

Ironically, the customers from whom I've received the highest praise in the past were those where something went wrong, sometimes really, embarrassingly wrong. But that’s not what they seemed to remember: What they wanted to talk about – for years, in some cases – was the amount and level of support the customer got when things did go wrong. What we heard from these customers was that they expect things to go wrong from time to time, sometimes seriously so. What they don’t expect is the vendor to pull out all the stops in order to fix whatever’s wrong – in some cases, whether it was our product or not.

Communication; Commitment; Competition; and Customers. If it sounds like a formula, it’s not. People, especially technical professionals, are complex, multi-dimensional and sometimes capable of extraordinary feats. Software development is a group endeavor and therefore a social process. Smart people thrive in an environment where they are surrounded by other smart people with a common goal, where their contributions are honored, where they’re treated with respect, and where they can have fun while they’re accomplishing great things.

Process

Processes should be defined that expedite team efforts rather than getting in the way. There are many aspects to this, but here I focus on agile software development methodologies.

Agile is the term used to refer to a class of development methodologies that are incremental and iterative. Scrum is by far the most popular agile method; others include XP (eXtreme Programming); Unified Process (UP, Agile UP, or Rational UP); Evo; Feature-Driven Development (FDD); Test-Driven Development (TDD); Dynamic Systems Development Method (DSDM); Crystal; and others. There is no “best” approach other than what works for a given team; in many cases, a hybrid is used (for example, Scrum + XP is widely used).

I have used Scrum in multiple settings, and have been impressed by it’s positive impact – especially where requirements are uncertain. For larger-scale developments I would be tempted to blend in aspects of Agile UP since it incorporates more SDLC artifacts into the process. In either case I think it’s important to incorporate test-driven development (TDD) concepts into the agile process.

Why do organizations adopt agile development methods? Generally, it’s for improved visibility to the project status from stakeholder, better adaptability in a dynamic environment, greater business value sooner in the development process, and overall risk reduction according to VersionOne as shown in the chart below:

While these are great reasons, I would say the biggest benefit is that engineers love it compared to traditional approaches, primarily because they are in much greater control over what they do and how they do it. And that’s a Good Thing: There’s no question in my mind that the best outcomes result from a team of well-motivated engineers who feel in control over their environment. But another key reason engineers love agile is it enables them to be heroes in customers eyes, since customer prize responsiveness more than anything. And agile allows development teams to be much more responsive to customer needs.

With Scrum, developers are fundamentally empowered to do their jobs in a team that collaborates together and makes its own technical decisions. According to Ken Schwaber, one of the Scrum founders and author of Agile Software Development With Scrum in 2002, Scrum teams are “self-managing, self-organizing, and cross-functional” and therefore control their own destiny.

But isn’t that like anarchy? Not at all. The Team (Capital T for a Scrum Team) is self-directed, but works from a prioritized Product Backlog that represents the interests of all stakeholders, including executive management. Scrum calls for three roles that work collaboratively to ensure alignment with business objectives and transparency to all stakeholders: Product Owner; ScrumMaster; and Team (typically 6-9 people, but can vary). The Product Owner represents customers’ requirements as well as requirements of other business stakeholders, and determines what are the highest priority features for each sprint (called the Sprint Backlog). On any given sprint it’s the Product Owner’s responsibility to ensure that the team is working on the highest priority items at that time. This is sometimes referred to as “just in time” requirements. And because requirements are dealt with on a just-in-time basis, feature creep and requirements uncertainty are minimized.

The agile project manager for Scrum is referred to as the ScrumMaster. He or she is responsible for ensuring that the Scrum process is followed, and for removing roadblocks as identified by Scrum Team members. Because the nature of agile project management is one of facilitation rather than top-down control, the ideal characteristics of an agile project management role such as ScrumMaster are those of a servant leader. Servant leadership, described as an executive model for large companies in the latter decades of the previous century, has now emerged as the preferred management style for agile software development.

The Scrum process involves one or more time-boxed sprints to a release (typical sprints are 4 weeks or 30 days). Ideally, each sprint should result in potentially releasable code – functionally complete (for the backlog items included in the sprint), refactored if necessary, tested, and documented. Within each sprint, there are daily scrum meetings (sometimes referred to as “stand-ups” – meetings so short and focused that they can be done standing up). At the end of a sprint, a review meeting takes place which usually includes a demo to the Product Owner and any interested stakeholders: One of the tenets of the Agile Manifesto is “working software is the primary measure of progress”.

The following diagram from TargetProcess shows the overall flow:

A variety of Scrum-specific project management tools exist, including from companies such as VersionOne, Atlassian, and Agilebuddy. These tools generally include dashboards for “at a glance” project status as well as burndown charts, defect tracking, velocity trends, analytics and a variety of reports.

Scrum – or for that matter, any agile methodology – can’t be plugged in to an organizational environment as a cookbook formula. The amount of what Craig Larman defines as "ceremony" in Agile and Iterative Development -- what we typically think of as structure (development phases), deliverables (artifacts of the development process), and process (workflow and authorizations) -- are unique to each development, based on dozens of factors. This requires wisdom and judgment, and, like the software being developed, should evolve incrementally and iteratively in every organization.

Summary

There is no single “best” management style; what works for any executive is a function of his or her values, character, and inner condition. In this post I have outlined some of the practices and leadership themes that have worked for me in over 20 years of executive experience.

Bartók

2010-07-06T09:39:00.000-07:00

Up until a year or so ago, I didn't know much about Béla Bartók other than what I had learned from doing crossword puzzles. Apparently the letters B-E-L-A are in pretty big demand. For example: Elba (clue "ere I saw..."). Or: Able (clue "...was I"). I had seen crossword puzzle clues such as "Hungarian Bartók", or "20th century composer Bartók" so I knew that there was a Bela (actually, Béla) Bartók who was a Hungarian composer in the 1900s.

I noticed about a year ago that I enjoyed music with "Hungarian" in the title by Haydn, Schubert, Brahms and others. But I wasn't sure what it meant, and I got curious. If I were to listen to Schubert's "German Dances" (D.820) followed by his "Hungarian Melody" (D.817), what precisely would I hear in the latter that made it Hungarian?

Questions such as that will ultimately lead one to Béla Bartók. Born in 1881, Bartók gained early fame as a virtuoso concert pianist in the center of western music -- Vienna. Home to Haydn, Mozart, Beethoven, Schubert, Brahms and Strauss, Vienna was the top destination for serious musicians. Bartók was awarded a scholarship in Vienna by the Emperor, but he shocked the music establishment by choosing instead to attend the Academy of Music in Budapest.

Bartók's intense interest in authentic Hungarian folk music is what kept him in Hungary. Working closely with his life-long friend and collaborator Zoltán Kodály, Bartók sought to establish a truly Hungarian national style. In order to do so, they decided to collect, catalog, and analyze authentic Hungarian folk music. How did they know it was authentic? They went out into rural Hungary, into the small towns and villages, and asked people to play for them for their recording device. (Bartók was a quiet reserved man, of an urban bent, usually fastidiously dressed. That he went out into these remote towns, and "let his hair down" in order to earn the trust of the villagers is testament to his strong interest.)

Bartók held this interest in folk music his entire life, which expanded from its initial focus on Hungarian peasant music to include Romanian, Slovak, Lithuanian, Polish, Russian, Turkish and even Arabic. One of Kodály and Bartók's initial findings: The so-called Hungarian music of earlier composers, referred to as the verbunkos style, was actually Gypsy music. Authentic folk music was far older, and was often played on native instruments that Bartók had never seen before. He found a similarity to ancient Greek music in these folk songs, to some extent because they are largely based on the pentatonic scale.

This was no passing interest. Bartók collected over 13,000 Hungarian folk songs in his lifetime; including the other ethnic strains, he (and Kodály) amassed over 20,000 folk songs. In doing so, what had previously been a strong nationalistic interest turned into a passion for music of the people, music that might bring nations together rather than drive them apart. His later compositional style was referred to as "Synthesis of East and West".

Bartók and Kodály were among most significant early figures in the field of ethnomusicology, the study of social and cultural aspects of music and dance in local and global contexts. Russian composers, led by Rimsky-Korsakov, were also trying to define and promote a true understanding of their native music at about the same time. One can hear the deep echoes of native lands in their music.

Here is what Bartók had to say about how he incorporated folk and peasant music into his compositions:

The question is, what are the ways in which peasant music is taken over and becomes transmuted into modern music? We may, for instance, take over a peasant melody unchanged or only slightly varied, write an accompaniment to it and possibly some opening and concluding phrases. This kind of work would show a certain analogy with Bach’s treatment of chorales. Another method is the following: the composer does not make use of a real peasant melody but invents his own imitation of such melodies. There is no true difference between this method and the one described above. There is yet a third way... Neither peasant melodies nor imitations of peasant melodies can be found in his music, but it is pervaded by the atmosphere of peasant music. In this case we may say, he has completely absorbed the idiom of peasant music which has become his musical mother tongue.

Bartók's post-romantic music doesn't appeal to everyone, especially to those not fortunate enough to have Hungarian blood flowing through their veins. But his unique blend of native folk music, rooted to the ancient lands, combined with the modern sound of the 20th century that was to bring unprecedented horror and dislocation, strikes a deep chord in our modern sensibilities.

Top Threats to Cloud Computing

2010-06-22T11:38:00.000-07:00

The CSA has recently issued a report called Top Threats to Cloud Computing in which they identify and discuss seven general threat areas:

Abuse and Nefarious Use of Cloud Computing
Insecure Application Programming Interfaces
Malicious Insiders
Shared Technology Vulnerabilities
Data Loss/Leakage
Account, Service and Traffic Hijacking
Unknown Risk Profile

No priority is implied in the ordering of the top threats; the advisory committee felt that further research and greater industry participation would be required to rank the threats. My view is that ranking is less important than applying a risk management discipline to the specific requirements of an organization considering cloud services.

As we consider the seven threats individually, we should keep in mind that the CSA considers this document as a first deliverable that will be updated regularly to reflect expert consensus on probable threats to cloud services:

Abuse and Nefarious Use of Cloud Computing
Because the Cloud Service Providers (CSPs) business model is based on rapid scalability, they have emphasized ease of adoption. Therefore, in most cases anyone with a valid credit card can register for and begin using cloud services in a matter of minutes. In other words, an attacker can materialize inside your CSP's infrastructure at any time, including on the same physical hardware your cloud-based application is running on, and you need to be prepared. The best policy is one of calculated paranoia: Assume your virtual environment includes all of your competitors as well as hackers, botnets, malicious users, clueless resource hogs, and other "nefarious users." Although as a user of cloud services you need to employ a layered defense strategy to protect critical resources, you also need to rely on your CSP's onboarding and technical surveillance processes: How effective is your CSP's registration and validation process for screening new users, and how well does your CSP's monitoring of internal traffic work?

Insecure Application Programming Interfaces
The same investor and market pressures that motivate CSPs to streamline the onboarding process also apply to how they support the configuration and use of their services by large numbers of users. The more these services can be enabled in a frictionless manner, the more profitable the CSP will be. Therefore, it's worth focusing on the APIs provided by CSPs for manging, deploying and maintaining cloud services. As the report points out, the "security and availability of general cloud services is dependent on the security of these basic APIs." Furthermore,

"From authentication and access control to encryption and activity monitoring, these interfaces must be designed to protect against both accidental and malicious attempts to circumvent policy."

One key question to ask: Does the CSP require use of X.509 certificates to access APIs? Besides being used to support the TLS protocol and WS-Security extensions to SOAP, X.509 certificates are used for code signing -- critical for secure use of APIs.

It's essential that users understand the security model of the CSP's APIs, especially to ensure that strong authentication and access controls are implemented.

Malicious Insiders
The threat from malicious CSP insiders is a threat that organizations have always had, except the threat was (and still is!) from someone they know rather than someone they don't know. An organization should compare its own policy with regard to insiders with that of the CSP, ensuring that controls such as the following exist:

State of the art intrusion detection systems
Background check on new hires (where permitted by law)
Authorized staff must pass two-factor authentication
Immediate deprovisioning of admin when no longer has business need
Extensive background check of staff with potential access to customer data
All admin access logged and audited, with suspicious actions raising a real-time alarm

Organizations should require transparency of CSP security and HR practices as well as all compliance reporting, and should refer to controls such as listed above as part of any legal agreement with the CSP.

Shared Technology Vulnerabilities
The foundation of the cloud service provider's business model is sharing of computing resources: CPU; memory; persistent storage; caches; and so forth. This sharing results in a multi-tenant environment, where great trust is placed in all virtualization technologies -- especially hypervisors that enable sharing of server hardware. Hypervisors must effectively isolate multiple guest operating systems while ensuring security and fairness. The CSA paper lists five remediation tactics for shared technology vulnerabilities, but the fact they they're generic recommendations (implement security best practices..., etc) serves to reinforce the point that at the end of the day, we need to be able to rely on the assumption that the CSP employs a secure hypervisor.

One potentially useful resource is a recently-released vSphere Security Hardening Guide from VMware. Overall, the guide contains more than 100 guidelines in a standardized format, with formally defined sections, templates, and reference codes that are in alignment with formats used by NIST, CIS, and others. The guide itself is split into the following major sections:

Introduction
Virtual Machines
Host
vNetwork
vCenter
Console OS (for ESX)

While the document is mostly applicable to CSPs using VMWare, many of the guidelines are generic and might apply to other hypervisors. In evaluating CSPs and shared technology vulnerabilities, it would be worthwhile having the CSP respond with how they've incorporated applicable recommendations from the hardening guide into their environment.

Data Loss/Leakage
The concept of defense in depth, or a layered security strategy, comes into play when we consider the threat of data loss or leakage. All of the above threat vectors can result in data loss or leakage. Data encryption, then, becomes the last line of defense against the data loss threat.

While encryption is easy enough conceptually, in practice it's a challenge -- especially in a multi-tenant environment. The authors of Cloud Security and Privacy dedicated an entire chapter to Data Security and Storage (as I previously discussed here). In particular, the authors warn of CSPs that use a single key to encrypt all customer data, rather than a separate key for each account (see pg. 69). Best practices for key management are provided in NIST's 800-57 "Recommendation for Key Management"; your CSP should comply or have an equivalent guideline that they use.

Of course you should know whether your CSP uses standard encryption algorithms, what they key length is, and whether the protocols employed ensure data integrity as well as data confidentiality. And since encrypted data at rest can't be operated on without being unencrypted you'll want to know whether memory, caches and temporary storage that have held unencrypted data are wiped afterward. The same set of questions (and answers) apply to the issue of data migration and to processes by which failed or obsolete storage devices are decommissioned.

Many regulatory frameworks focus on protecting against data loss and leakage. If you need to comply with PCI DSS or any other set of financial controls you will need to ensure adequate threat protection that includes encryption of data at rest.

Account, Service and Traffic Hijacking
In the online payment space there's a segment called "card not present" (CNP). That's analogous to cloud computing, where service is provided to a "user not present". All of the threats in an enterprise environment -- including phishing, fraud, shared or stolen credentials and weak authentication methods -- become magnified in the cloud. Remediation suggestions are fairly obvious: prohibit sharing of credentials; leverage strong two-factor authorization where possible; employ proactive monitoring to detect unauthorized activity; and understand CSP security policies and SLAs. I would add to CSA's recommendations that organizations should routinely check for excessive access rights to ensure there are no unused (and unmonitored) accounts that would be vulnerable to highjacking.

Unknown Risk Profile
CSPs, hypervisor vendors, other cloud technology providers, application developers, security experts, and customers are all pushing the envelope when it comes to cloud services. The compelling economics of cloud services are driving adoption rates higher than is typical for new technologies. All together, this adds an element of technical uncertainty to the question of what are the top threats to cloud computing.

In general, a strategy of pragmatic paranoia is recommended. Be on the alert for the unexpected. Review logs, set up monitoring and alerting systems where practical, and re-evaluate the security implications of your cloud service periodically. Most importantly, select a CSP you can trust and back it up with a strong agreement specifying all areas of concern and including SLAs -- with penalties for non-compliance.

Productivity Tools

2010-06-15T16:24:00.000-07:00

I've done a fair amount of research to find apps the span my laptop, my netbook and my iPhone to keep key information in sync and readily available any time, on any device. Based on my own usage patterns, preferences and quirks, these are my primary organization and multi-device sync apps: Nozbe, Evernote, GoodReader, Reeder and Seesmic. Here's how I use them (your mileage may vary):

Nozbe
I've been searching for the perfect to-do list manager/project manager/everything manager for years. In my opinion such a system must support a comprehensive approach to organization and prioritization that addresses personal, professional and project-oriented activities, both short-term and longer-term. I've found a system that works for me, called GTD -- Getting Things Done. The app I use, Nozbe, is based on the GTD system. The best way to get up to speed on GTD is to read the book by the GTD guru, David Allen. It's an easy read; I got through my library copy in a couple days. There are numerous GTD articles and blogs as well.

Nozbe is a web-based system for managing and prioritizing tasks and projects, and so it's equally accessible no matter which computer I use. There's an iPhone app for when I'm away from my computer. For tasks that have a date component, Nozbe links to Google Calendar. My primary selection criterion for Nozbe was its integration with Evernote, my information organization app.

Evernote

Evernote keeps everything I want to remember organized and available: whole web pages and clips from selected web pages; meeting and call notes (Evernote is now my default word processor); voice memos; pictures, drawings, whiteboard snapshots and technical diagrams; emails; and PDF files. My information is stored on the Evernote web site and is available from my two PCs and my iPhone (Evernote is also available on BlackBerry and Android).

Evernote items can be organized into separate notebooks, and each note can be assigned tags. You can search by notebook, by tag, by date created, and by location created (great for when you enter things from your iPhone; if you can remember what city you were in when you created a note, you can find it).

Evernote provides browser plug-ins (I use the "clip to Evernote" plug-in on both Chrome and Firefox). For any web page you're on, you can select text, click on the plug-in icon, and your selected text is saved into Evernote along with the URL of the site. Imagine being able to save and later be able to retrieve anything of interest you come across on the web! I use this feature a lot, and don't know how I lived without it.

Evernote also allows you to import a folder from your hard drive and include it in a specified notebook. As files are added or deleted to the folder, Evernote is automatically updated. This is particularly invaluable for project work, in that you can keep your notes, web reference items and documents organized by project. Furthermore, it's all accessible from the iPhone. Many people like the Dropbox application for syncing files to their iPhone, but Evernote is because you can select specific sub-folders without having to reorganize your computer's file system.

Here's a pretty good article about the many uses of Evernote.

GoodReader
One of the issues with having PDF files available on the iPhone is the small display makes them hard to read if not unreadable, depending on the formatting. That's why I complement Evernote with GoodReader. It has a capability called PDF reflow that word-wraps the text from a PDF file to fit the iPhone screen legibly. Even very large PDF docs can be read using GoodReader. While it requires a manual step to pick those PDF files you may want to read when you're away from your PC, it's not too painful to do.

Reeder
One of my primary means to keep track of everything that's going on in selected areas of interest is Google Reader. I've set up Google Reader to follow many blogs of interest (271 as of this writing). Google Reader supports folders, so blogs can be organized by company, topic or project. I've completely eliminated the web surfing I used to do to major news and technical sites, and instead use Reader to access the data on my own schedule, knowing that I won't miss anything because Reader will keep it until I get to it, keeping track of what I've read and what I haven't. Also, Evernote is integrated with Google Reader so when I find an article of interest, I simply send it to Evernote. The best iPhone client for Google Reader I've found is Reeder; it's probably the app I use most often on my iPhone. Reeder isn't integrated with Evernote (yet) so when I find an article or blog post I want to keep while in Reeder I simply "star" it, and next time I'm on my PC I can then send it to Evernote. (Another option from Reeder would be to send the article as an email to my Evernote account.)

Seesmic
I had used the official Twitter iPhone app until I found Seesmic. Why? Besides the ability to view Twitter and Facebook feeds from a single app, the fact that Seesmic is integrated with Evernote so important tweets or Facebook updates can be saved (assuming I would ever see an important FB update).

As you can see, an underlying theme of my productivity apps is their integration with Evernote, allowing me to establish a universal information store in the cloud that can be accessed from any computer or smart phone. In my view, Evernote fulfills the original promise of Microsoft's OneNote by being open and well integrated outside of the MS Office family.

One last note: I was surprised that I couldn't sync my Google Contacts over the air when I got my iPhone out of the box. I found an app that does it very well, it's called SyncInABlink.

So what is the practical result of this? The last trip I took, I left my notebook and laptop at home and only took my iPhone. The iPhone -- with the right productivity apps -- is fully adequate for staying connected. For meetings, phone calls and general brainstorms I just make notes into Evernote, and with email, Reeder and Seesmic I stay fully informed of all relevant news and developments. Of course if I had any presentations to develop or documents to write I'd take a computer; otherwise all I need is my iPhone.

______________________________________________

GoodSync
For those who may be interested in how to sync multiple computers. I went from one computer to two last year when my laptop died. Two pieces of good news: first, I backup on a nightly basis, and the hardware failure occurred first thing in the morning -- no data loss; second, I got the ThinkPad warranty when I bought the unit new and it was still covered. However, I had an investor presentation in two days and the return/repair would take about a week. So I went out and bought a netbook for $300, loaded all my files onto it, and made it through the week (but I unfortunately didn't get funded).

My first challenge: How to keep the netbook and laptop in sync. I had recently upgraded to the latest version of iTunes that allowed me to sync my wife's and my music (she has a Mac). Apple, as usual, had thought through the consumer experience and wireless sync over iTunes worked flawlessly -- just like you'd expect. With that model in mind, I looked for a general purpose tool for syncing all files wirelessly between my ThinkPad and netbook (Acer Aspire One). I couldn't find one; instead, I found apps that could sync to/from a USB drive. The best of the lot (I tested three) was GoodSync; it worked, had the sync options I wanted, was reasonably priced, and discounted subsequent licenses.

I have three sync profiles set up on my laptop: my daily backup, which runs automatically every day and propagates deletes so that the backup mirrors the primary; a versioned backup, which doesn't propagate deletes and which saves prior versions of changed/deleted files; and a third profile to sync to the USB drive (which I run when I've changed any files on my netbook, say when I use it for travel). I have a separate copy of GoodSync on my netbook, which I run each time before I'm going to take my netbook with me -- and again when I return if I've modified any files.

The Growth of Web Services

2010-06-14T10:37:00.000-07:00

eBay published their first web API in 2000. It took another 8 years to get to 1,000 APIs on the web; it only took 18 months to get to the next 1,000 APIs.

ProgrammableWeb was founded in 2005, when they tallied 105 APIs. The current count is 2,016 and the rate of new APIs is doubling year over year.

What segments account for these APIs? Social networking sites are high on the list, followed by mapping, financial, reference and shopping. The single most popular API is Google Maps, used in 1,978 mashups.

Even more dramatic are the stats for how often APIs are called. Here's the Internet's new billionaire club:

74% of the APIs are REST and 15% are SOAP; the remainder includes JavaScript, XML-RPC and AtomPub. Over the past two years the use of REST APIs has increased as an overall percentage of net APIs, mostly at the expense of SOAP. Another trend is the increasing use of JSON; 45% of all new APIs support JSON. And on the authentication front, OAuth continues to pick up steam as over 80 APIs now have OAuth support.

The web is evolving from providing access to information, to providing access to services, to providing access to complex services -- also known as mashups. The popular and somewhat trivial example is the number of sites that call Google Maps API to show a map to their location. Links to Flickr, YouTube and Twitter are also popular. But what is the real business potential of these complex services?

APIs enable further leverage in systems development; that's why we can think of the web as a platform. Object-oriented software development is giving way to service oriented architecture (SOA) , which allows interfaces to be specified and their web services to be made available to any system with web access. This allows development organizations to focus on their core competencies, and leverage web services for the rest.

An example of how this is playing out is in monetizing the web. A new generation of web-based services has emerged, and many of these services are based on subscription revenue models rather than single transactions (aka shopping carts). Subscription billing is hard: while it's tempting for the many new developments in digital publishing, gaming, telecommunications, health care, consumer electronics and renewable energy to include a do-it-yourself billing system, there's no need to. Companies such as Zuora, Vindicia and Aria Systems provide sophisticated billing systems through APIs, providing advanced functionality such as currency conversation, tax calculation, invoicing, fraud control, collections, reporting and analytics for a fraction of the time and expense that it would take to self-develop such capabilities. As we evolve towards a subscription economy with a variety of payment models, APIs providing web billing services will be leveraged to ensure secure, reliable billing.

User Activity Monitoring

2010-06-08T09:09:00.000-07:00

Gartner recommends that organizations implement user activity monitoring as part of a strategy to manage external and internal threats, and for regulatory compliance. Gartner suggests integrating Identity and Access Management (IAM) capabilities with a SIEM system to achieve user activity monitoring, but other approaches work as well if not better as I explain below.

Why is user activity monitoring needed? Since all major regulatory frameworks -- including SOX, PCI DSS, GLBA, and HIPAA -- require least privilege access controls, thousands of companies are obligated to prevent excessive access rights and yet, according to Deloitte, have failed to adequately do so. The reason this is a hard problem has to do with the dynamic nature of the enterprise-especially in an economic downturn -- with layoffs, restructurings, aggressive use of contractors and other service providers, along with the need for federated identity and access management as enterprises collaborate.

Conventional wisdom holds that the best practice for resolving this issue is to adopt an IAM system with role-based access control (RBAC) capabilities. Unfortunately, such systems provide no user activity monitoring or other assessment mechanisms and as a result are notoriously ineffective. While these systems ensure that only authorized users may log in to critical resources, they fail to consistently determine which users should be authorized to access those resources. As a result, as reported by a Dartmouth field study and by IDC, over-entitlement is the norm. In many organizations over 50% of access rights are dormant, representing a huge security vulnerability as well as a significant compliance exposure.

This is where user activity monitoring comes in. Organizations can assess user privileges, or entitlements, through user activity monitoring in order to identify excess entitlements. That few organizations do so is indicated by the high rate of audit findings for such access controls. Two additional methods of implementing user activity monitoring, besides the SIEM+IAM integration suggested by Gartner, are network-based activity monitoring and log-based activity monitoring.

Many organizations collect NetFlow data for IP traffic analysis reasons, and analyze this data for user activity monitoring. While NetFlow shows source and destination IP address and port number, it doesn't show authenticated user names nor application names (applications can in many cases be deduced with destination IP address and port number, but it's practically impossible to link source IP address to user names). NetFlow is therefore inadequate in most cases for tracking user access to audited applications.

Some organizations have adopted a network-based user activity monitoring system which goes beyond NetFlow to record, not just source and destination IP addresses, but authenticated user names and which application was accessed. While far superior to a NetFlow-only approach, network based activity monitoring has several challenges:

Span port scarcity - span ports are used for a variety of applications, and without a network monitoring system such as one from Gigamon span port availability could be a constraint;
Span port data loss - most switches are vulnerable to packet loss on their span ports during peak traffic bursts. Even a data loss rate of under 1% can render such a solution inadequate for forensic purposes;
Application-side scalability - network activity monitoring requires a probe on every ingress span into the application infrastructure;
User-side scalability - a probe must be placed in every subnet with its own AD or other authorization system, which can make for a very expensive deployment in a distributed environment or one with many remote offices;
Encryption - as the percentage of encrypted sessions inside the data center increases, it leaves a larger blind spot for network-based approaches;
Technical challenges with today's DPI silicon in monitoring 10G links - the latest generation network processor with DPI (deep packet inspection ) capabilities can monitor 4-5 Gbps, far short of the 20 Gbps required for full-duplex traffic monitoring of a 10G link; and
No visibility to access from behind the monitored span port - network activity monitoring is blind to local access, e.g. from the application server's console port. It also can't see application-to-application access.

Despite these challenges, enterprises are deploying network-based access activity monitoring system because they otherwise do not have effective solutions for preventing excessive access rights.

An alternate approach to network-based access activity monitoring is log-based user activity monitoring, also known as Identity and Access Assessment (IdAA), which does not suffer from the limitations and constraints listed above. Cloud Compliance, my prior company, read log files for audited applications in order to prevent excessive access rights and other access audit violations. The log-based approach precludes the need for hardware to be deployed, is scalable, detects 100% of access activity (regardless of encryption, 10G links, and source of access) and, when deployed as a SaaS solution, eliminates the need for installation, software maintenance, and a large upfront capital outlay.

Visualizing Security Metrics

2010-06-01T08:10:00.000-07:00

This is the third and final post discussing Security Metrics: Replacing Fear, Uncertainty and Doubt by Andrew Jaquith. As I noted, Jaquith makes some intriguing and vital points about the need for "good" metrics and "serious analytic scrutiny" to inform executive decision-making on issues of security, compliance, and risk governance. This is an especially important topic today, with organizations everywhere trying to figure out how to stay secure and improve compliance while cutting their expense budget.

Most organizations, when considering appropriate investment levels to deal with risk, are not lacking for data. But lots of data does not equate to relevant information required for sound decision-making. Jaquith's point is that information in the form of metrics -- good metrics, which he defines -- is lacking in many enterprises.

But once good metrics have been defined, how are they communicated to stakeholders? Jaquith dedicates an entire chapter to visualization. He starts by listing his six design principles for visualization of metrics:

It is about the data, not the design (resist urges to "dress up" the data)

Just say no to three-dimensional graphics and cutesy chart junk (it obscures your data)

Don't go off to meet the wizard (or talking paperclips)

Erase, erase, erase (removing tick marks and grid lines results in a crisp chart with few distracting lines)

Reconsider Technicolor (default colors are far too saturated, and should be muted. Consider a monochromatic palette)

Label honestly and without contortions (pick a meaningful title, label units of measure, don't abbreviate to the point where the meaning is not clear)

Like me, Jaquith is an admirer of Edward Tufte, author of several books about information visualization including the classic The Visual Display of Quantitative Information (1983, Cheshire, CT: Graphics Press). According to Tufte, a key to effective visual displays is understanding the goal of your presentation. In Tufte's own words:

At the heart of quantitative reasoning is a single question: Compared to what? Small multiple designs, multivariate and data bountiful, answer directly by visually enforcing comparisons of changes, of the differences among objects, of the scope of alternatives. For a wide range of problems in data presentation, small multiples are the best design solution.

Hence, we have small multiples as a visualization strategy. Here's an example:

From this display, one can look at different categories (in this case, departments) to view comparative performance over time. Once can readily imagine security/compliance applications for this approach, such as dormant accounts by resource, or excessive access rights by department.

In his book Beautiful Evidence (2006, Cheshire, CT: Graphics Press) Tufte introduces a refinement to this concept called the sparkline, which he defines as "small, intense, simple datawords". The example Tufte uses to explain the sparkline concept is a patient's medical data, taken from Beautiful Evidence:

Besides Tufte's small multiples and sparklines, Jaquith's visualization suggestions include indexed and quartile time series charts, bivariate charts, period-share charts, treemaps, and Pareto charts. The key point is that there's not a single graphic approach that works in all cases; one needs to determine the essence of what is being conveyed. The audience almost always consists of busy people, often executives, who need to have information presented clearly and in context. It doesn't do anyone any good to be able to point out after a security event that the "smoking gun" data had been seen, but it was either lost in the noise of too much data, or its significance was not clear.

P.S. It's not necessarily relevant to this post, but my favorite graphical display of quantitative information is an advertisement for one of Tufte's books that regularly appears in Scientific American and The Economist:

More Security Metrics

2010-05-26T10:08:00.000-07:00

Although I wrote about Security Metrics: Replacing Fear, Uncertainty and Doubt by Andrew Jaquith earlier, a single post doesn't do this important topic justice. The key theme as expressed by Jaquith is

...information security is one of the few management disciplines that has yet to submit itself to serious analytic scrutiny.

This lack of analytic scrutiny in the form of security metrics makes risk management especially difficult for executive understanding and guidance, especially when discussing the necessary level of investment required. Executives ideally want their security and compliance metrics to answer the following questions:

How effective are my security processes?
Am I better off than I was this time last year?
How do I compare with my peers?
Am I spending the right amount of money?
What are my risk transfer options?

As previously discussed, most functions within an enterprise -- HR, finance, manufacturing, supply chain, call center, e-commerce and operations -- have the ability to measure their performance by tracking key metrics, and comparing with other companies in a peer group. Such metrics share the characteristics of being simple to explain, readily lending themselves to benchmarking, and being consistently and automatically collected.

Without such metrics, we're doomed to reactive rather than proactive risk management. Or, as Jaquith calls it, we're on the hamster wheel of pain:

Here are Jaquith's suggested questions for management when measuring audit and compliance processes and their related investments:

How much time and effort are security staff spending on audit-related activities? (Metrics: # regulatory audits completed, time/cost of audit activities)
Have audits uncovered serious weaknesses in existing controls? (Metrics: % security compliance reviews with material weaknesses, % key external requirements compliant per external audit)
How much time and effort are security staff spending fixing problems uncovered by audits? (Metrics: # pending deficiencies and estimated time/cost to complete, time/cost spent on remediation activities)
Have audit activities uncovered problems with controls that would affect customer trust or privacy? (Metric: # pending customer-related deficiencies and estimated time/cost to complete)

Only by employing security metrics and submitting to serious analytic scrutiny can an enterprise get security and compliance risk management off of the hamster wheel of pain and onto a level playing field with other disciplines.

Security Metrics

2010-05-20T09:42:00.000-07:00

Andrew Jaquith, in his book Security Metrics: Replacing Fear, Uncertainty and Doubt, describes the value of metrics in general and in doing so identifies one of the key challenges in ensuring system security:

Today's information security battleground is all about entitlements -- who's got them, whether they were granted properly, and how to enforce them.

The book describes how metrics can be applied in managing security systems in general, and in entitlements/access rights in particular. Jaquith, a senior analyst at Forrester, cites examples of how other disciplines and industries use key metrics to compare their operations to peer companies. For example, freight companies know their freight cost per mile and loading factors-as well as those of their competitors. Management can therefore set meaningful objectives and measure themselves against comparable companies. Choosing to be above, on, or below an industry average is a question of strategy as well as operational efficiency. For example, a freight company may be willing to have a lower load factor than its peers if that's the tradeoff required to offer faster delivery times (for which it presumably charges a premium).

Similarly, warehousing firms measure and compare their cost/square foot and inventory turns, and e-commerce companies measure their website conversion rates. And of course financial metrics have been standardized and reported on for years. Companies can therefore compare relevant metrics to those of their peers in order to better evaluate their internal performance.

Could such a use of metrics apply to security? And can metrics be of use in the "entitlements battleground"?

First, let's look at Jacquith's definition of a good metric:

consistently measured, without subjective criteria;
cheap to gather, preferably in an automated way;
expressed as a cardinal number or percentage, not with qualitative labels such as high, medium and low;
expressed using at least one unit of measure, such as "defects" or "dormant accounts"; and
contextually specific -- relevant enough to decision-makers so that they can take action.

So what about the "information security battleground", namely entitlements and access rights? What metrics are relevant to that? Jaquith lists pertinent questions and the metrics that can guide management actions, for example: Does the organization review employee entitlements? An example metric would be % accounts dormant. (The complete discussion starts on page 117 of Jaquith's book under the heading Ensuring System Security.)

One of the advantages of a multi-tenant SaaS solution is the global statistical perspective that can be provided, which allows customers to compare their performance to that of their peers. By knowing industry averages for key metrics customers can benchmark their internal performance and security objectives to those of comparable organizations. What better way to arm oneself for the information security battleground known as entitlements management?

The definition and application of security metrics is ongoing. One resource I recommend is Securitymetrics.org, which provides empirical strategies for decision-makers and security practitioners and which includes links to digests, presentations, and handouts from past Metricon Workshops.

Is Perfect Access Control Possible?

2010-05-11T17:14:00.000-07:00

Bruce Schneier, the Chief Security Technology Officer of BT and a highly regarded security guru, engaged in a point/counter-point debate with Marcus Ranum in an article entitled Schneier-Ranum Face-Off: Is Perfect Access Control Possible?

The question regarding the efficacy of access controls is particularly relevant today, especially in light of the fact that excessive access rights was the top audit finding over the past two years. How can that be resolved? The general consensus among Identity Management (IdM) experts is that organizations should implement a role-based access control (RBAC) system to manage access rights. But as Schneier points out:

RBAC is very hard to implement correctly. Organizations generally don't even know who has what role. The employee doesn't know, the boss doesn't know--and these days the employee might have more than one boss -- and senior management certainly doesn't know.

Ranum notes that part of the problem is that we're paying for decisions made over the past decade to make critical data easier and cheaper to access.

What both Schneier and Ranum agree on is that over-entitlement is the norm today, and these excessive access rights -- also called excessive entitlements -- represent a security and compliance exposure.

So where does that leave us? Based on what I've seen, I have to agree with Schneier's assessment:

In the end, a perfect access control system just isn't possible; organizations are simply too chaotic for it to work.

If RBAC systems are so hard to implement correctly, and even if doing so still leaves the organization with excessive access rights and their associated risks and vulnerabilities, what can be done? User activity monitoring in the form of an Identity and Access Assessment (IdAA) solution can complement RBAC identity management systems by providing feedback that uncovers excess entitlement in the form of dormant (aka zombie) accounts. Therefore, even if RBAC is very hard to implement correctly, at least the organization can gain visibility into and remove the vulnerabilities and compliance exposure associated with excessive access rights.

Cloud Security and Privacy

2010-05-04T09:23:00.000-07:00

I wanted to discuss a newly-published book, Cloud Security and Privacy: An Enterprise Perspective on Risks and Compliance. But the book has too much valuable content to do it justice in a 500-word blog post, so I will focus today on a single chapter: Data Security and Storage.

First, props to the authors (Tim Mather, Subra Kumaraswarmy, and Shahed Latif) who have written a thoughtful, in-depth book on a topic that's often subject to hype and relatively unsatisfying sound bites. The authors aren't working an agenda, and they aren't promoting cloud services. Nor do they provide easy answers. But they do offer insights as to what the critical issues are, what questions to ask your cloud service provider (CSP) to truly assess relevant risk factors, and what strategies might be considered when your security and privacy requirements exceed the service levels provided by current cloud services.

In my discussions with customers, the biggest concerns I hear relative to public cloud services are on the subject of data security, especially data privacy and data remanence.The authors discuss aspects of data security related to data in transit and data at rest, including multitenancy issues. Here's a partial checklist: You should know whether your CSP uses vetted encryption algorithms, and whether the protocols employed ensure data integrity as well as data confidentiality. You should be aware that even when data at rest is encrypted, it can't be operated on by the application without being unencrypted -- in such a case you'll want to know whether memory, caches and temporary storage are wiped afterward (the answer is almost certainly "no", or, more likely, such questions won't be answered by the CSP). The same set of questions (and answers) apply to the issue of data migration and to processes by which failed or obsolete storage devices are decommissioned.

The key point in this chapter is with regard to data security mitigation. How can you compensate when CSP data security capabilities are inadequate to your needs? The authors' answer: Don't put sensitive data in a public cloud, other than for simple cloud storage services where your data is (and always remains) encrypted. I couldn't agree more, although I would add that this is an area that CSPs are aware of and working on, and I predict that in the near future (2-3 years) public cloud data security will have improved substantially.

A prerequisite to evaluating whether public CSPs' security is adequate to your needs is to classify your data. Only by doing so can an organization make informed judgments as to whether the cloud security is "good enough". The organization's policy should be to limit cloud-based applications to only those that operate on low- or moderate-risk data, such as CRM and internal log data. Higher-risk data sets may be stored in the public cloud only if they have been "sanitized" (i.e. sensitive data removed or anonymized).

Beethoven

2010-04-23T11:42:00.000-07:00

I've been reading a biography of Beethoven, and came across a remarkable passage that I thought was worth sharing. The year is 1810; a prominent music critic, E. T. A. Hoffmann, has reviewed Beethoven's Fifth Symphony in the Allgemeine Musikalische Zeitung where he calls it "one of the most important works of the time". Hoffmann then attempts to set Beethoven in context:

In Haydn's compositions the expression of a youthful, light-hearted spirit is dominant. His symphonies lead us into an infinite green grove, in a cheerful, gaily colored throng of merry people. Mozart leads us into the depths of the spiritual world. Fear grips us, but without torment; it is more a foreboding of the eternal... Beethoven's instrumental music also opens up to us the world of the immense and infinite. Glowing rays of light blaze through the dark night of this world and we are made conscious of gigantic shadows which surge up and down, gradually closing in on us more and more annihilating everything within us, except the torment of endless longing...Beethoven bears deep within his nature the romantic spirit of music, which he proclaims in his works with great genius and presence of mind. Your reviewer has never felt this so clearly as in this particular symphony which, more than any other of his works, unfolds Beethoven's romantic spirit in a climax rising straight to the end and carries the listener away irresistibly into the wondrous spirit world of the infinite.

Beethoven himself said of his muse:

You will ask me where I get my ideas. That I cannot tell you with certainty; they come unsummoned, directly, indirectly -- I could seize them with my hands -- out in the open air, in the woods, while walking, in the silence of the night, early in the morning, incited by moods, which are translated by the poet into words, by me into tones that sound, and roar and storm about me until I have set them down into notes.

The Problem with Entitlements and Access Controls

2010-04-23T09:22:00.000-07:00

Ronald Reagan famously said "Trust, but verify". He could very well have been talking about entitlement management systems, which manage authorization to critical applications and other IT resources. Such systems are trusted to maintain control over entitlements (also called privileges or access rights). However, the systems themselves rarely have verification or assessment capabilities. This may be adequate for smaller organizations or enterprises where roles change infrequently. But the dynamic nature of most enterprises -- with layoffs, restructurings, aggressive use of contractors and other service providers -- makes assessment not only prudent, but necessary to ensure effective access controls and audit compliance.

Entitlements

Deloitte, in The 6th Annual Global Security Survey, reports that excessive entitlements, also known as excessive access rights, was the top audit finding over the past year -- for the second year in a row! In other words, a fundamental access control that represents a compliance exposure and security vulnerability was the top audit finding in 2007 and, despite all the attention that garnered, was also the top audit finding in 2008 (the latest year for which survey data exist).

Since all major regulatory frameworks, including SOX, PCI DSS, GLBA, NERC and HIPAA, require access controls, many thousands of companies are obligated to prevent excessive access rights and yet, according to the Deloitte survey, have failed to effectively do so.

Not only is excessive access rights the top audit finding, but IDC states that such vulnerabilities result in major financial exposure -- and that up to 60% of rights on most systems are expired and therefore dormant. The problem is that IT and security staff at most companies don't know that dormant accounts exist -- or more precisely, they suspect they exist but don't know how to find or remediate them.

Why is this a hard problem to solve?

Access Controls in the Real World

A paper written by a team at Dartmouth describes observations from field study research of both retail and investment banks. The study was more in-depth than most surveys we hear about; for example, the study team was embedded for three weeks in the security group of an investment bank. The report focuses primarily on internal access controls and the risks of over-entitlement, and they directly address the challenge of effectively managing access controls.

What they found was that the frequent shifting of staff may from one department or role to another often results in users accumulating entitlements over time. Part of the problem is this: Entitlement management systems assume that an employee's direct supervisor can make informed decisions about what entitlements are required to do their job. But as the Dartmouth team points out:

"As more organizations take on a matrix structure, it becomes less evident who reports to whom and who is responsible for permitting and terminating data access."

This leads to ambiguous and unwieldy structures for assigning entitlements, or privileges, as shown in Figure 1:

Figure 1: Privileging in traditional hierarchical corporate structures (left) vs. in dynamically, "matrixed" organizations (right). An arrow represents a supervising relationship (directed graph). Note that on the left, each person has exactly one direct supervisor, whereas on the right, each may have two or more.

And even if the corporate structure and reporting relationship is clear in all cases, the degree of scale and complexity makes entitlement management a big problem as shown in Figure 2:

Figure 2: Complexity and dynamicism in entitlement systems. The number of applications, entitlements and users make it a large-scale problem, and the number of daily modifications makes it a fast-moving target.

The biggest challenge isn't the massive number of entitlements and users, however, but the highly dynamic nature of employees and organizational structure within the firm.

Conventional wisdom holds that role-based access control (RBAC) systems are the answer. By allowing organizations to segregate the massive numbers of employees and entitlements into work groups, RBAC systems make the entitlement management process more effective. But the size, complexity and dynamic nature of many large enterprises make role-based access control challenging, to say the least. Quoting from the Dartmouth study:

"At one very large retail bank that we interviewed, the CISO had recently completed an RBAC project creating 11,000 roles across the firm to control access to nearly 22,000 applications. Developing the roles took a team two years and the ongoing review process was expected to be significant."

In the real world, access rights are constantly changing, for legitimate reasons: employees are hired and terminated; contractors come and go; service providers and outsource firms require access on a project basis with often unclear timelines; federated identity management systems expand the concept of trusted user beyond the enterprise boundary; departments and whole companies undergo reorganizations; mergers and acquisitions result in major restructurings; layoffs lead to rapid and sometime undocumented role changes; and employees transferring within a company inevitably have to overlap responsibilities (and access) between their old and new jobs. Unclear and imperfect communications between HR, line-of-business (LOB) staff, and IT exacerbate the problem.

Managing Entitlements

Andrew Jaquith, an analyst at Forrester, in his book Security Metrics states:

"Today's information security battleground is all about entitlements-who's got them, whether they were granted properly, and how to enforce them."

Companies large and small employ different approaches to entitlement management, with equal lack of success. Mostly, they do manual reviews of entitlements prior to audits by going through HR records, reviewing application logs, and interviewing LOB managers-a process inevitably referred to as a fire drill. Other approaches to entitlement management include development of custom reports for SEIM and log management systems, network-based user activity monitoring, and RBAC systems.

The management challenge is to determine what's a reasonable target level of excessive access rights in terms of percentage of overall rights granted, and then ensure that solutions are in place to consistently keep actual excessive access rights on or below the target. It's more expensive to establish an excessive access rights target of 2% than of 4%, for example. Therefore, management must determine what level constitutes "enough" security, doesn't break the budget or put an undue burden on IT or line-of-business staff, and yet meets the compliance requirements as measured by auditors. What auditors are looking for is a sustainable, measureable process that demonstrates visibility (can the company detect when and where it has excessive access rights?) and the ability to remediate problems when they occur (can the company eliminate excessive access rights within a reasonable amount of time from their detection?).

Top IT Audit Findings

2010-04-13T07:37:00.000-07:00

In a prior post I referred to the 2008 Deloitte survey which reported that excessive access rights have been the top audit finding for each of the past two years. Of all the security-related issues that IT auditors investigate, excessive access rights -- also known as over-entitlements, or failure to maintain least privilege -- was the most common vulnerability uncovered. Here's a chart showing the top 8 internal/external audit findings for 2007 and 2008, ranked by percentage of respondents citing findings in each category, with a brief explanation how my prior company, Cloud Compliance, would have addressed each issue:

Excessive access rights. Note that despite the improvement from 2007, excessive access rights remained the top audit finding in 2008 as reported in an earlier post. Part of the reason that excessive access rights has been the top finding for the past two years is that auditors have raised the standard, from evidence of the existence of a process to evidence that the process is effective. Due to the urgency of this issue, and the lack of effective solutions available, this was an initial focus of Cloud Compliance.

Segregation of duties. Segregation of duties, also referred to as separation of duties and abbreviated SoD, is one of the most fundamental concepts of security and control, and also one of the most difficult to achieve. Cloud Compliance's innovative 3-layer rights model enabled definition of benchmark rights, where SoD concepts are embodied. Our analytics can report on inconsistencies between benchmark rights, provisioned rights and actual rights as detected by access activity in order to assure continued compliance with key segregation of duty principles.

Access control compliance with procedures. This audit issue is closely related to excessive access rights; access control is required to prevent users without appropriate rights from accessing audited resources. Cloud Compliance's Identity and Access Assessment (IdAA) solution was able to determine if access controls were effective.

Lack of audit trails/logging, lack of documentation of controls, and lack of review of audit trails. I'm grouping these three top findings together because they represent the facet of access audit where technology and process come together. Application logs, which represent the most effective way to determine user access activity, are an essential tool for ensuring that access controls are compliant. And reports that list who has access to what, along with who should have access to what, become critical components of how access controls are documented.

Excessive developers' access to production systems and data. This audit finding is challenging to address, because it's unrealistic in most operating environments to completely block developers from accessing production systems for troubleshooting and critical maintenance operations. The objective, then, is not to prevent such access but to note when it's risen to an "excessive" level. Cloud Compliance addressed this by allowing a policy to be defined where a reasonable max level of developer access to production systems could be specified, along with a lower threshold for an early warning system. Access levels could then be compared to historical equivalents for trend analysis as well.

Lack of clean-up of access rules following a transfer or termination. There's a clever vendor that claims to "take the SH out of IT". One of the reasons that there's an SH in IT in the first place is the typical IT department's need to manage rights and access rules in a real-world environment with re-org, restructurings, layoffs, role re-definitions and transfers. Especially transfers. Because transfers are not a discrete event so much as a process where an employee has overlapping responsibilities between new job and old job-and therefore must maintain access rights for both jobs. And the duration of the overlap can't be determined in advance. Cloud Compliance's advanced analytics examined user activity to determine when a user's rights to resources required for a previous role could be de-provisioned -- which ideally would be before an auditor happened to discover excessive access rights.

My prior company, Cloud Compliance, developed an Identity and Access Assessment (IdAA) solution to address the top IT audit findings as reported by Deloitte. As noted above, our initial solution helped organizations eliminate excess entitlements (also called dormant accounts, or zombie accounts). We identified users with excess entitlements, and provided tools for isolating high levels of over-entitlement by group, business unit or by application. Unfortunately, although we validated customer demand and the lack of competing solutions, we were unable to raise venture capital to scale the company.

Excessive Access Rights

2010-04-05T07:34:00.000-07:00

Deloitte, in The 6th Annual Global Security Survey, reports that excessive access rights was the top "internal/external audit finding over the past 12 months" -- for the second year in a row.

What is meant by "excessive access rights", why is it important, and why did it remain the top audit finding in 2008 after all the attention it drew by being the top audit finding in 2007? In other words, why is this a hard problem to solve?

A cornerstone of security best practices -- and therefore of compliance requirements -- is to limit access to critical resources to only those employees and users who have a legitimate business need to access those resources. As a result, most companies adopt a policy of "least privilege" which is intended to restrict users to access only those applications that are required to do their job. See the table below for the relevant least privilege text in each of the major regulatory frameworks:

Whereas least privilege is the best practice, excessive access rights result from failing to achieve an idealized implementation of least privilege. And in the real world, completely eliminating excessive access rights is practically impossible.

The management challenge is to determine what's a reasonable target level of excessive access rights in terms of percentage of overall rights granted, and then ensure that solutions are in place to consistently keep actual excessive access rights on or below the target. And the tradeoff in establishing a "reasonable" target is -- you guessed it -- cost. It's more expensive to establish an excessive access rights target of 2% than of 4%, for example. Therefore, management must determine what level constitutes "enough" security, doesn't break the budget or put an undue burden on IT or line-of-business staff, and yet meets the compliance requirements as measured by auditors. What auditors are looking for is a sustainable, measurable process that demonstrates visibility (can the company detect when we have excessive access rights) and the ability to remediate problems when they occur (can the company eliminate excessive access rights within a reasonable amount of time from their detection).

Why is this so hard?

Companies large and small that we have talked to employ different approaches to this issue, with equal lack of success. Mostly, they do manual reviews prior to audits going through HR records, reviewing application logs, and interviewing LOB managers -- a process consistently referred to as a fire drill. Other approaches include development of custom reports for SEIM and log management systems, and network-based user activity monitoring.

As the Deloitte survey reports -- and customers confirm -- current approaches have failed to achieve the desired and necessary level of compliance. My prior company, Cloud Compliance, was founded to address this specific problem. Unfortunately, we were unable to raise venture funding. And so, as far as I know, there are no available solutions to comprehensively address this issue.