Rational Survivability

[Disclaimer: I’m not a QSA. I don’t even play one on the Internet. Those who are will generally react to posts like these with the stock “it depends” answer, to which I respond “you’re right, it does.  Not sure where that leaves us other than with a collective sigh, but…]

The Payment Card Industry (PCI) last week released version 2.0 of the Data Security Standard (DSS.) [Legal agreement required]  This is an update from v1.2.1 but strangely does not introduce any major new requirements but instead clarifies language.

Accompanying this latest revision is also a guidance document titled “Navigating PCI DSS: Understanding the Intent of the Requirements, v2.0” [PDF]

One of the more interesting additions in the guidance is the direct call-out of virtualization which, although late to the game given the importance of this technology and its operational impact, is a welcome edition to this reader.  I should mention I’ve sat in on three of the virtualization SIG calls which gives me an interesting perspective as I read through the document.  Let me just summarize by saying that “…you can’t please all the people, all of the time…” 😉

What I find profoundly interesting is that since virtualization is a such a prominent and enabling foundational technology in IaaS Cloud offerings, the guidance is still written as though the multi-tenant issues surrounding cloud computing (as an extension of virtualization) don’t exist and that shared infrastructure doesn’t complicate the picture.  Certainly there are “cloud” providers who don’t use infrastructure shared with other providers beyond themselves in order to deliver service to different customers (I think we call them SaaS providers,) but think about the context of people wanting to use AWS to deliver services that are in scope for PCI.

Here’s what the navigation document has to say specific to virtualization and ultimately how that maps to IaaS cloud offerings.  We’re going to cover just the introductory paragraph in this post with the guidance elements and the actual DSS in a follow-on.  However, since many people are going to use this navigation document as their first blush, let’s see where that gets us:

PCI DSS requirements apply to all system components. In the context of PCI DSS, “system components” are defined as any network component, server or application that is included in, or connected to, the cardholder data environment. System components” also include any virtualization components such as virtual machines, virtual switches/routers, virtual appliances, virtual applications/desktops, and hypervisors.

I would have liked to see specific mention of virtual storage here and although it’s likely included by implication in the management system/sub-system mentions above and below, the direct mention of APIs. Thanks to heavy levels of automation, the operational movements related to DevOps and with APIs becoming the interface of the integration and management planes, these are unexplored lands for many.

I’m also inclined to wonder about virtualization approaches that is not server-centric such as physical networking devices, databases, etc.

If virtualization is implemented, all components within the virtual environment will need to be identified and considered in scope for the review, including the individual virtual hosts or devices, guest machines, applications, management interfaces, central management consoles, hypervisors, etc. All intra-host communications and data flows must be identified and documented, as well as those between the virtual component and other system components.

It can be quite interesting to imagine the scoping exercises (or de-scoping more specifically) associated with this requirement in a cloud environment.  Even if the virtualized platforms are operated solely on behalf of a single customer (read: no shared infrastructure — private cloud,)  this is still an onerous task, so I wonder how — if at all — this could be accomplished in a public IaaS offering given the lack of transparency we see in today’s cloud operators.  Much of what is being asked for relating to infrastructure and “data flows” between the “virtual component and other system components” represents the CSP’s secret sauce.

The implementation of a virtualized environment must meet the intent of all requirements, such that the virtualized systems can effectively be regarded as separate hardware. For example, there must be a clear segmentation of functions and segregation of networks with different security levels; segmentation should prevent the sharing of production and test/development environments; the virtual configuration must be secured such that vulnerabilities in one function cannot impact the security of other functions; and attached devices, such as USB/serial devices, should not be accessible by all virtual instances.

“…clear segmentation of functions and segregation of networks with different security levels” and “the virtual configuration must be secured such that vulnerabilities in one function cannot impact the security of other functions,” eh? I don’t see how anyone can expect to meet this requirement in any system underpinned with a virtualized infrastructure stack (hardware or software) whether it’s multi-tenant or not.  One vulnerability in the hypervisor makes this an impossibility.  Add in management, storage, networking. This basically comes down to trusting in the sanctity of the hypervisor.

Additionally, all virtual management interface protocols should be included in system documentation, and roles and permissions should be defined for managing virtual networks and virtual system components. Virtualization platforms must have the ability to enforce separation of duties and least privilege, to separate virtual network management from virtual server management.

Special care is also needed when implementing authentication controls to ensure that users authenticate to the proper virtual system components, and distinguish between the guest VMs (virtual machines) and the hypervisor.

The rest is pretty standard stuff, but if you read the guidance sections (next post) it gets even more fun.  This is why the subjectivity, expertise and experience of the QSA is so related to the quality of the audit when virtualization and cloud are involved.  For example, let’s take a sneak peek at section 2.2.1, as it is a bit juicy:

2.2.1 Implement only one primary function per server to prevent functions that require different security levels from co-existing
on the same server. (For example, web servers, database servers, and DNS should be implemented on separate servers.)
Note: Where virtualization technologies are in use, implement only one primary function per virtual system component.

I  acknowledge that there are “cloud” providers who are PCI certified at the highest tier.  Many of them are SaaS providers.  Many simply use their own server stacks in co-located facilities but due to their size and services merely call themselves cloud providers — many aren’t even virtualized per the description above.   Further, there are also methods of limiting scope and newer technologies such as tokenization that can assist in solving some of the information-centric issues with what would otherwise be in-scope data, but they offset many of the cost-driven efficiencies marketed by mass-market, low-cost cloud providers today.

Love to hear from an IaaS public cloud provider who is PCI certified (to the VM boundary) with customers that are in turn certified with in-scope applications and cardholder data or even a SaaS provider who sits atop an IaaS provider…

Just read this first before responding, please.

/Hoff

Related articles

• “PCI DSS 2.0 standard released, but detailed virtualization guidance still unavailable” and related posts (itknowledgeexchange.techtarget.com)
• Dear Verizon Business: I Have Some Questions About Your PCI-Compliant Cloud… (rationalsurvivability.com)
• Scoping Fun with PCI DSS 2.0 (brandenwilliams.com)
• PCI DSS 2.0 Released, Updating Requirements for Credit Card Security (palisadesystems.com)
• What’s The Problem With Cloud Security? There’s Too Much Of It… (rationalsurvivability.com)

You’ll forgive my impertinence, but the last time I saw a similar claim of a PCI compliant Cloud offering, it turned out rather anti-climatically for RackSpace/Mosso, so I just want to make sure I understand what is really being said.  I may be mixing things up in asking my questions, so hopefully someone can shed some light.

This press release announces that:

“…Verizon’s On-Demand Cloud Computing Solution First to Achieve PCI Compliance” and the company’s cloud computing solution called Computing as a Service (CaaS) which is “…delivered from Verizon cloud centers in the U.S. and Europe, is the first cloud-based solution to successfully complete the Payment Card Industry Data Security Standard (PCI DSS) audit for storing, processing and transmitting credit card information.”

It’s unclear to me (at least) what’s considered in scope and what level/type of PCI certification we’re talking about here since it doesn’t appear that the underlying offering itself is merchant or transactional in nature, but rather Verizon is operating as a service provider that stores, processes, and transmits cardholder data on behalf of another entity.

Here’s what the article says about what Verizon undertook for DSS validation:

To become PCI DSS-validated, Verizon CaaS underwent a comprehensive third-party examination of its policies, procedures and technical systems, as well as an on-site assessment and systemwide vulnerability scan.

I’m interested in the underlying mechanicals of the CaaS offering.  Specifically, it would appear that the platform – compute, network, and storage — are virtualized.  What is unclear is if the [physical] resources allocated to a customer are dedicated or shared (multi-tenant,) regardless of virtualization.

According to this article in The Register (dated 2009,) the infrastructure is composed like this:

The CaaS offering from Verizon takes x64 server from Hewlett-Packard and slaps VMware’s ESX Server hypervisor and Red Hat Enterprise Linux instances atop it, allowing customers to set up and manage virtualized RHEL partitions and their applications. Based on the customer portal screen shots, the CaaS service also supports Microsoft’s Windows Server 2003 operating system.

Some details emerge from the Verizon website that describes the environment more:

Every virtual farm comes securely bundled with a virtual load balancer, a virtual firewall, and defined network space. Once the farm is designed, built, and named – all in a matter of minutes through the CaaS Customer Management Portal – you can then choose whether you want to manage the servers in-house or have us manage them for you.

If the customer chooses to manage the “servers…in-house (sic)” is the customer’s network, staff and practices now in-scope as part of Verizon’s CaaS validation? Where does the line start/stop?

I’m very interested in the virtual load balancer (Zeus ZXTM perhaps?) and the virtual firewall (vShield? Altor? Reflex? VMsafe-API enabled Virtual Appliance?)  What about other controls (preventitive or detective such as IDS, IPS, AV, etc.)

The reason for my interest is how, if these resources are indeed shared, they are partitioned/configured and kept isolated especially in light of the fact that:

Customers have the flexibility to connect to their CaaS environment through our global IP backbone or by leveraging the Verizon Private IP network (our Layer 3 MPLS VPN) for secure communication with mission critical and back office systems.

It’s clear that Verizon has no dominion over what’s contained in the VM’s atop the hypervisor, but what about the network to which these virtualized compute resources are connected?

So for me, all this all comes down to scope. I’m trying to figure out what is actually included in this certification, what components in the stack were audited and how.  It’s not clear I’m going to get answers, but I thought I’d ask any way.

Oh, by the way, transparency and auditability would be swell for an environment such as this. How about CloudAudit? We even have a PCI DSS CompliancePack 😉

Question for my QSA peeps: Are service providers required to also adhere to sections like 6.6 (WAF/Binary analysis) of their offerings even if they are not acting as a merchant?

/Hoff

Related articles by Zemanta

• PCI DSS Compliance and IT Security: Harmony or Discord? (prweb.com)
• Brief PCI Council Interview in Regards to PCI DSS 2.0 (chuvakin.blogspot.com)
• Revisions to Credit Card Security Standard on the Way (pcworld.com)
• Data Encryption for PCI 101: Introduction (securosis.com)
• Why your QSA should not be your Security Partner (brandenwilliams.com)
• Ask HN: Are you PCI DSS compliant? (pcisecuritystandards.org)
• Can You Have a PCI Compliant Virtualized Web Site? (securecloudreview.com)

I kicked off a bit of a dust storm some months ago when I wrote a tongue-in-cheek post titled "Please Help Me: I Need a QSA to Assess PCI/DSS Compliance In the Cloud."  It may have been a little contrived, but it asked some really important questions and started some really good conversations on my blog and elsewhere.

Cloud Sites, Mosso|The Rackspace Cloud’s Flagship offering, is officially the very first cloud hosting solution to enable an Internet merchant to pass PCI Compliance scans for both McAfee’s PCI scans and McAfee Secure Site scans. 

This achievement occurred just after Computer World published an article where some CIO’s shared their concern that Cloud Computing is still limited to “things that don’t require full levels of security.”  This landmark breakthrough may be the beginning of an answer to those fears, as Mosso leads Cloud Hosting towards a solid future of trust and reliability.

Mosso's blog featured an example of a customer — The Spreadsheet Store — who allegedly attained PCI compliance by using Mosso's offering. Pay very close attention to the bits below:

“We are making the Cloud business-ready.  Online merchants, like The Spreadsheet Store can now benefit from the scalability of the Cloud without compromising the security of online transactions,” says Emil Sayegh, General Manager of Mosso|The Rackspace Cloud.  “We are thrilled to have worked with The Spreadsheet Store to prepare the Cloud for their online transactions.”

…

The Spreadsheet Store set up their site using aspdotnetstorefront, “Which is, in our opinion, the best shopping cart solution on the market today,” says Murphy.  “It also happens to be fully compatible with Mosso.”  Using Authorize.Net, a secure payment gateway, to handle credit card transaction, The Spreadsheet Store does not store any credit card information on the servers.  Murphy and team use MaxMind for fraud prevention, Cardinal Commerce for MasterCard Secure Code and Verified by Visa, McAfee for PCI and daily vulnerability scans, and Thawte for SSL certification.

Cloud Sites is not currently designed for the storage or archival of credit card information.  In order to build a PCI compliant e-commerce solution, Cloud Sites needs to be paired up with a payment gateway partner.

The fact that Mosso is seeking ways to help their customers off-load as much PCI compliance requirements to other 3rd parties is fine – it makes business sense for them and their merchant customers.  It’s their positioning of the effort as a “landmark breakthrough” and that they are somehow pioneers which leads to generalisations rooted in misunderstandings that is the problem.
Next time you hear someone say ‘Cloud Provider X is PCI compliant’, ask the golden PCI question: is their Cloud receiving, processing, storing or transmitting Credit Card data (as defined by the PCI DSS)?  If they say ‘No’, you’ll know what that really means…marketecture.

I'm happy to say that there appears to be some good news on the PCI DSS front with the promise of a SIG being formed this year for virtualization.  This is a good thing. 

You'll remember my calls for better guidance for both virtualization and ultimately cloud computing from the council given the proliferation of these technologies and the impact they will have on both security and compliance.

In that light, news comes from Troy Leach, technical director of the PCI Security Standards Council via a kind note to me from Michael Hoesing:

This is a good first step.  if you've got input, make sure to contribute!

/Hoff

Hello.

I wonder if you might help me.

I operate an e-commerce Internet-based business that processes and stores cardholder data.

I need a QSA to assess my infrastructure and operations for PCI/DSS compliance.

Oh, I forgot to mention.  All my infrastructure is in the cloud.  It's all virtualized.  It runs on Amazon's EC2.  All my data is hosted outside of my direct stewardship.  I don't own anything.

Since the cloud hides all the infrastructure and moving parts from me, I don't know if I meet any of the following PCI requirements:


I don't know if there are firewalls. I don't know about the cloud-vendor's passwords, AV, access control/monitoring, vulnerability management or security processes.

A friend told me about section 12.8, but it doesn't really apply because the "service" provider just provides me cycles and storage, I run the apps I build but I don't see any of the underlying infrastructure.

Also, I have no portability for BCP/DR because my AMI only runs on the Amazon cloud, nowhere else.  I don't know who/how backups are done outside of my manifest.

I'm sure we could ask though, right?

Seems logical…

However, this is the quandary we're facing with virtualization and cloud computing.  In terms of the companies that hire these PCI compliance experts, the assessment methodology/requirements are predicated upon a "standard" that continues to be out of touch with the economic and technological world around it.  That's not the experts' fault, they're scoring you against a set of requirements that are black and white. 

As companies try and leverage technology to be more secure, to transfer risk, to focus on the things that matter most and reduce costs — if you believe the marketing — It's really a no-win situation.

The PCI Security Standards Council doesn't even have a SIG for virtualization and yet we see the crushing onslaught of virtualization with no guidance and this tidal wave has been rushing at us for at least 3-5 years.   If you believe the uptake of cloud computing, we're blindly hurdling over the challenges that virtualized internally-owned infrastructure brings and careening headlong down a path to cloud computing that leaves us in non-compliance.

The definition of what a "service provider" means and how they interact with the cardholder data companies are supposed to protect needs to be redefined.

It's time the PCI Council steps up and gets in front of the ball and not crushed by it such that the companies that would do the right thing — if they knew what that meant — aren't punished by an out-of-touch set of standards.

Another interesting example I use in my VirtSec presentations when discussing the challenges of what I describe as Phase 2 of virtualization — virtualizing critical applications and things like Internet-facing infrastructure in DMZ’s — is the notion of compliance failures based on existing and upcoming revisions to regulatory requirements.

Specifically, I use PCI/DSS to illustrate that in many cases were one to take a highly-segmented and stratified "defense-in-depth" architecture that is today "PCI compliant" and virtualize it given presently available options, you’d likely find yourself out of compliance given the current state of technology solutions and auditing standards used to assess against.

Then again, you might just pass with flying colors while being totally insecure.

Here’s a fantastic example from Eric Siebert over at the TechTarget Virtualization blog.  Check this out, it’s a doozie!

Having just survived another annual PCI compliance audit, I was again surprised that the strict standards for securing servers that must be followed contain nothing specific concerning virtual hosts and networks. Our auditor focused on guest virtual machines (VMs), ensuring they had up-to-date patches, locked-down security settings and current anti-virus definitions. But ironically, the host server that the virtual machines were running on went completely ignored. If the host server was compromised, it wouldn’t matter how secure the VMs were because they could be easily accessed. Host servers should always be securely locked down to protect the VMs which are running on them.

It seems that much of the IT industry has yet to react to the virtualization trend, having been slow in changing procedures to adjust to some of the unconventional concepts that virtualization introduces. When I told our auditor that the servers were virtual, the only thing he wanted to see was some documentation stating that the remote console sessions to the VMs were secure. It’s probably just a matter of time before specific requirements for virtual servers are introduced. In fact, a recent webinar takes up this issue of whether or not virtualized servers can be considered compliant, addressing section 2.2.1 of the PCI DSS which states, “Implement only one primary function per server”; that is to say, web servers, database servers and DNS should be implemented on separate servers. Virtual servers typically have many functions running on a single physical server, which would make them noncompliant.

So let’s assume that what Eric talks about in section 2.2.1 of PCI/DSS holds true, that basically means two things: (1) PCI/DSS intimates that virtualization cannot provide the same level of security as non-virtualized infrastructure and (2) you won’t be able to virtualize infrastructure governed by PCI/DSS if you expect to be compliant.

Now, this goes toward the stuff Mogull and I were talking about in terms of assessing risk and using the notion of "zone defense" for asset segmentation in virtualized infrastructure. 

Here’s a snippet from my VirtSec preso on the point:

Further, as I mentioned in my post titled "Risky Business — The Next Audit Cycle: Bellweather Test for Critical Production Virtualized Infrastructure," this next audit cycle is going to be interesting for many companies…

Yippeee!

/Hoff

A few days ago, Topps Meat company, a 67-year old company and one of the largest producers of frozen meat products in the country, shut its doors for good.

Why?

They had a breach of the sanitation persuasion.  From the NY Times:

Topps Meat Company, one of the country’s largest manufacturers of
frozen hamburgers, said today it was going out of business after it
recalled more than 21.7 million pounds of ground beef products last
month.

The company, based in Elizabeth, N.J., said a few of its 87 employees will remain at the plant to help the United States Department of Agriculture investigate how the E. coli bacteria tainted frozen hamburger patties made there.

Anthony D’Urso, the chief operating officer at Topps, said the company
was unable to withstand the financial burden of the recall.

“This is tragic for all concerned,” Mr. D’Urso said in a statement. “In
one week we have gone from the largest U.S. manufacturer of frozen
hamburgers to a company that cannot overcome the economic reality of a
recall this large.”

On Sept. 25, the United
States Department of Agriculture announced a recall of frozen hamburger
patties from Topps, saying that the meat was potentially tainted by E.
coli bacteria. Officials at the agency conceded that they knew that
meat from Topps was contaminated on Sept. 7, when the first positive
test results for E. coli came back.

The financial strain associated with a recall of spoiled meat in a single week killed them.

So what does this have to do with data breaches?

When the ChoicePoint scandal hit, we saw Card Services shutter due to direct economic pressure (they could no longer process credit cards) brought about by the fallout from data breaches, but contrast that with the experience of a recent "breacher" such as TJX and some might argue that not only has it not actively impacted their P&L negatively, but it’s made them a better, stronger and more profitable company.  The figures don’t lie:

After the TJX debacle I remember seeing predictions that people will vote with their feet. Of course they didn’t, sales actually went up 9%. The same argument was made for Ruby Tuesdays who lost some credit cards. It just doesn’t happen. Lake Chad and disasters on a global scale continue to plague us due to climate change yet still people refuse to stop buying SUV’s.

Check out the chronology of security breaches from the Privacy Rights Clearinghouse.  The total
            number of records containing sensitive personal information involved in security breaches:
          167,308,738

That number is mounting every day
and some of these breaches you don’t even hear about in the press.
Have we become so
desensitized to this breach fiasco that it’s become just a mild
inconvenience?  Or is it that credit card number losses have been subconsciously classified outside of the scope of "identity theft?"

Think about it.  Having your credit card number stolen is really, in the scope of things, not that big of a deal.  You call the CC company, dispute any charges you didn’t make, they close the account and despite the inconvenience, that’s it.  Then a new card shows up in the mail, sometimes with a larger spending limit!  Sweet!

The liability is minimal.  It’s happened to me twice.  My credit wasn’t impacted, my life didn’t end.  In fact, I got a card with a cool Koi on it that matches one of my tattoos.   I’m not saying it goes that "well" for everyone, but what’s the impetus for consumer outrage?

As soon as the liability is shifted away from the banks who suck it up and take the hit (as do the vendors whose merchandise is stolen,) and moves closer to the consumer,  we’ll see some agitation and consumer outrage.

Until then, I suppose we’re content to just go on eating spoiled meat (as it were) and get a new credit card number every three months until a company like Topps — or rather one that people really care about — goes through the meat grinder and closes its doors.

Where’s the beef?

/Hoff

I read the Harvard Business Review frequently and find that the quality of writing and insight it provides is excellent.  This month’s (September 2007) edition is no exception as it features a timely data breach case study written by Eric McNulty titled "Boss, I think Someone Stole Out Customer Data."

The format of the HBR case studies are well framed because they ultimately ask you, the reader, to conclude what you would do in the situation and provide many — often diametrically opposed — opinions from industry experts.

This month’s commentators were Bill Boni (CISO, Motorola,) James E. Lee (SVP ChoicePoint,) John Coghlan (former President & CEO of Visa,) and Jay Foley (Executive Director of the Identity Theft Resource Center)

The fictitious company profiled is Flayton Electronics, a regional electronics chain with 32 stores across six states.  The premise of the fictitious data breach focuses on the manner in which Flayton Electronics decides what to do, how to interact with LEO, and how/if to communicate the alleged data breach consisting of potentially thousands of their customer’s credit cards. 

What I liked about the article are the classic quote gems that highlight the absolute temporal absurdity of PCI compliance and the false sense of security it provides to the management of companies — especially in response to a breach.

You know, "We’re compliant, thus we’re secure, ergo we’re at less risk."

Now, I’m not suggesting that compliance initiatives don’t make things "better," in some sense, but they don’t necessarily make a company more "secure."  I think the case study demonstrates that well enough and the readership of this blog certainly doesn’t need to be convinced.

So, why write about it then?  The quote snippets below illustrate reality — sometimes hysterically.  You’ll have to read the entire story to gain true context and to appreciate the angst this sort of thing brings, but I chuckled a couple of times when reading these quotes:

“What’s our potential exposure?” Brett inquired matter-of-factly.
Quietly he wondered whether the firm’s PCI compliance would provide
sufficient protection.

“Why do we have to notify customers at all?” Brett asked, genuinely
puzzled. “Haven’t the banks already informed them that their accounts
have been compromised?”

“What about some kind of coincidence?” Brett was grasping at straws.
“Perhaps 1,500 of our customers just had the same bad luck?”

“We’re still trying to determine what happened,” the CIO offered meekly.

“But we are sure that our PCI systems were working, right?” Brett pushed.

“Becoming PCI compliant is
complicated,” Sergei hedged, “especially when you’re constantly
improving your own technology.” He ran through a laundry list of the
complexities of recent improvements. At any given moment, Sergei had
three or four high-priority tech projects in various stages of
implementation. It was a constant juggling act.

Brett, in a rare display of anger, pounded his fist on Sergei’s
desk. “Are you saying, Sergei, that we’re not actually PCI compliant?”

Sergei stiffened. “We meet about 75% or so of the PCI requirements.
That’s better than average for retailers of our size.” The response was
defensive but honest.

“How have we been able to get away with that?” Brett growled. He
knew that PCI compliance, which was mandated by all the major credit
card companies, required regular scans by an outside auditor to ensure
that a company’s systems were working—with stiff penalties for failure.

“They don’t scan us every day,” Sergei demurred. “Compliance really is up to us, to me, in the end.”

Sergei reported finding a hole—a disabled firewall that was supposed to
be part of the wireless inventory-control system, which used real-time
data from each transaction to trigger replenishment from the
distribution center and automate reorders from suppliers.

“How did the firewall get down in the first place?” Laurie snapped.

“Impossible to say,” said Sergei resolutely. “It could have been
deliberate or accidental. The system is relatively new, so we’ve had
things turned off and on at various times as we’ve worked out the bugs.
It was crashing a lot for a while. Firewalls can often be problematic.”

Sounds like a typical Monday morning staff meeting to me…I think you could be a fly in the wall in many mid-size (or large, for that matter) companies and hear this same set of quotes — regardless of how many millions of dollars the company may have spent on compliance initiatives.  It is indeed sad to see how many of these folks don’t realize that "compliance" is merely the floor, not the ceiling.  <sigh>

If you pay close attention to the dynamics of the management team within the story, you’ll bear witness to all seven distinct stages of the data breach grieving process:

• Shock or Disbelief

• Denial

• Bargaining

• Guilt

• Anger

• Depression

• Acceptance and Hope

I’m not really aiming for a punchline here, but I will suggest that you read the entire story to appreciate the tale in the grandest of its context.  The commentary from the industry experts is also very interesting…

/Hoff

P.S. I think it’s very cool the HBR allows you to access these stories without paying or registering and allows one to use up to 500 words on blogs and the like for the non-commercial purpose of summarizing the story.  Nice policy.