Rational Survivability

This post contains nothing particularly insightful other than a pronounced giant sucking sound that's left a vacuum in terms of forward motion regarding security and virtualization.

Besides the sumo suit wrestling match I'm organizing between myself and Simon Crosby at this year's coming RSA 2009 show, I'm really excited to announce that there will be another exciting virtualization security (VirtSec) event happening at the show.

"In this verbal cage match session, two well known critics of virtualization security take on two virtualization company CTOs as they spar over how best to secure virtualization platforms: who should be responsible for securing it, and how that ultimately impacts customers and attackers.  We have Hoff and Skoudis versus Crosby and Herrod.  Refereeing will be respected analyst, Antonopoulos."

I just read the latest blog of Alistair Croll from GigaOm titled "Cloud Security: The Sky Is Falling!" in which he suggests that we pillow-hugging security wonks ought to loosen our death grips on our data because not only are we flapping our worry feathers for nothing, but security in "the Cloud" will result in better security than we have today. 

As "the Cloud" provider adds customers, the capability to secure the infrastructure and the data transiting it, ultimately becomes an issue of scale, too. The more automation that is added, the more false positives show up, especially in light of the fact that the service provider has little or no context of the information, business processes or business impact that their monitoring tools observe.  You can get rid of the low-hanging fruit, but when it comes down to impacting the business, some human gets involved.

Read more…

I'm working on the sequel to my Four Horsemen of the Virtualization Security Apocalypse presentation.

It's called "The Frogs Who Desperately Wanted a King: An Information Security Fable of Virtualization, RTI and Cloud Computing Security." (Okay, it also has the words "interpretive dance" in it, but that's for another time…)

Many of the interesting issues from the Four Horsemen regarding the deficiencies of security solutions and models in virtualized environments carries over directly to operationalizing security in the Cloud. 

As a caveat, let's focus on a cost-burdened "large" enterprise who's involved in moving from physical to virtual to cloud-based services.

I'm not trying to make a habit of picking on Amazon AWS, but it's just such a fantastic example for my point, which is quite simply:

Why?  Let me paint the picture…

In non-virtualized environments, we generally use dedicated appliances or integrated solutions that provide one of more discrete security functions. 

These solutions are built generally on hardened OS's, sometimes using custom hardware and sometimes COTS boxes which are tarted up.  They are plumbed in between discretely segregated (physical or logical) zones to provide security boundaries defined by arbitrary policies based upon asset classification, role, function, access, etc.  We've been doing this for decades.  It's the devil we know.

In virtualized environments, we currently experience some real pain when it comes to replicating the same levels of security, performance, resiliency, and scale using software-based virtual appliances to take the place of the hardware versions in our physical networks when we virtualize the interconnects within these zones.

There are lots of reasons for this, but the important part is realizing that many of the same hardware solutions are simply not available as virtual appliances and even when they are, they are often not 1:1 in terms of functionality or capability.  Again, I've covered this extensively in the Four Horsemen.

So if we abstract this to its cloudy logical conclusion, and use AWS as the "platform" example, we start to face a real problem for an enterprise that has a decade(s) of security solutions, processes and talent that is focused on globalizing, standardizing and optimizing their existing security infrastructure and are now being forced to re-evaluate not only the technology selection but the overall architecture and operational model to support it.

Now, it's really important that you know, dear reader, that I accept that one can definitely deploy security controls instantiated as both network and host-based instances in AWS.  There are loads of options, including the "firewall" provided by AWS.

However, the problem is that in the case of building an AMI for AWS supporting a particular function (firewall, WAF, IPS, IDS, etc.) you may not have the same solutions available to you given the lack of support for a particular distro, lack of "port" to a VA/VM, or issues surrounding custome kernels, communication protocols, hardware, etc…  You may be limited in many cases to having to rely on open source solutions.

In fact, when one looks at most of the examples given when describing securing AWS instances, they almost always reference OSS solutions such as Snort, OSSEC, etc.  There's absolutely NOTHING wrong with that, but it's only one dimension.

That's going to have a profound effect across many dimensions.  In many cases, enterprises have standardized on a particular solution not because it's special from a "security" perspective, but because it's the easiest to manage when you have lots of them and they are supportable 24/7 by vendors with global reach and SLA's that represent the criticality of their function.

That is NOT to say that OSS solutions are not easy to manage or supportable in this fashion, but I believe it's a valid representation of the state of things.

(Why am I getting so defensive about OSS? 😉

Taking it further, and using my favorite PCI in the Cloud argument, what if the web application firewall that you've spent hundreds of thousands of dollars purchasing, tuning and deploying in support of PCI DSS throughout the corporate enterprise simply isn't available as a software module installable in an AMI in the cloud?  Or the firewall?  Or the IPS? 

In the short term this is a real problem for customers.  In the long term, it's another potential life preserver for security ISV's and an opportunity for emerging startups to think about new ways of solving this problem.

/Hoff

Update: Ooops.  I forgot to announce that I'm once again putting on my Devil's Advocacy cap. It fits nicely and the contrasting color makes my eyes pop.;)

It should be noted that obviously I recognize that dedicated
hardware offers performance and scale capabilities that in many cases
are difficult (if not impossible) to replicate in virtualized software
instantiations of the same functionality. 

However, despite spending the best part of two years raising
awareness as to the issues surrounding scalability, resiliency,
performance, etc. of security software solutions in virtualized
environments via my Four Horsemen of the Virtualization Security Apocalypse presentation, perception is different
than reality and many network capabilities will simply consolidate into the virtualization platforms until the next big swing of the punctuated equlibrium.

This is another classic example of "best of breed" versus "good enough" and in many cases this debate becomes a corner-case argument of speeds and
feeds and the context/location of the network topology you're talking
about. There's simply no way to sprinkle enough specialized hardware around to get the pervasive autonomics across the entire fabric/cloud without a huge chunk of it existing in the underlying virtualization platform or underlying network infrastructure.

THIS is the real scaling problem that software can address (by penetration) that specialized hardware cannot.

There will always be a need for dedicated hardware for specific needs, and if you have an infrastructure service issue that requires massive hardware to support traffic loads until the sophistication and technology within the virtualization layer catches up, by all means use it!  In fact, just today after writing this piece Joyent announced they use f5 BigIP's to power their IaaS cloud service…

In the longer term, however, application delivery control (ADC) will ultimately become a feature of the virtual networking stack provided by software as part of a larger provisioning/governance/autonomics challenge provided by the virtualization layer.  If you're going to get as close to this new atomic unit of measurement in the VM, you're going to have to decide where the network ends and the virtualization layer begins…across every cloud you expect to host your apps and those they may transit.

I've been reading Lori McVittie's f5 DevCentral blog for quite some time.  She and Greg Ness have been feeding off one another's commentary in their discussion on "Infrastructure 2.0" and the unique set of challenges that the dynamic nature of virtualization and cloud computing place on "the network" and the corresponding service layers that tie applications and infrastructure together.

The interesting thing to me is that why I do not disagree that that the infrastructure must adapt to the liquidity, agility and flexibility enabled by virtualization and become more instrumented as to the things running atop it, much of the functionality Greg and Lori allude to will ultimately become a function of the virtualization and cloud layers themselves*.

One of the more interesting memes is the one Lori summarized this morning in her post titled "Managing Virtual Infrastructure Requires an Application Centric Approach," wherein the she lays the case for the needs of infrastructure becoming "application" centric based upon the "highly dynamic" nature of virtualized and cloud computing environments:

…and yet the applications themselves, despite how integrated they may be, suffer from the same horizontal management problem as the network today does.  So I'm not so sure about the finality of the "it's all about the application" because we haven't even solved the "virtual infrastructure management" issues yet.

Bridging the gap between where we are today and the infrastructure 2.0/application-centric focus of tomorrow is illustrated nicely by Billy Marshall from rPath in his post titled "The Virtual Machine Tsunami," in which he describes how we're really still stuck being VM-centric as the unit measure of application management:

So we still have all the problems of managing the applications atomically, but I think there's some general agreement between these two depictions.

However, where it gets interesting is where Lori essentially paints the case that "the network" today is unable to properly provide for the delivery of applications:

This is where I start to squint a little because Lori's really taking the notion of "application intelligence" and painting what amounts to a router/switch in an appliction delivery controller as a "platform" as she attempts to drive wedge between an ADC and "the network."

Besides the fact that "the network" is also rapidly evolving to adapt to this more loosely-coupled model and the virtualization layer, the traditional networking functions and the infrastructure service layers are becoming more integrated and aware thanks to the homgenizing effect of the hypervisor, I'll ask the question I asked Lori on Twitter this morning:

Why won't this ADC functionality simply show up in the hypervisor?  If you ask me, that's exactly the goal.  vCloud, anyone?  Amazon EC2?  Azure?

If we take the example of Cisco and VMware, the coupled vision of the networking and virtualization 800 lb gorillas is exactly the same as she pens above; but it goes further because it addresses the end-to-end orchestration of infrastructure across the network, compute and storage fabrics.

So, why do we need yet another layer of network routers/switches called "application delivery controllers" as opposed to having this capability baked into the virtualization layer or ultimately the network itself?

So, can you really make the case for deploying more "application-centric" routers/switches (which is what an application delivery controller is) regardless of how aware it may be?

/Hoff

It's Thanksgiving here in the U.S., so in between baking, roasting and watching Risk Astley rickroll millions in the Macy's Thanksgiving Day Parade, I had a thought about how the utility and agility of the cloud computing models such as Amazon AWS (EC2/S3) and the pricing models that go along with them can actually pose a very nasty risk to those who use the cloud to provide service.

That thought — in between vigorous whisking during cranberry sauce construction — got me noodling about how the pay-as-you-go model could be used for nefarious means.

Specifically, this usage-based model potentially enables $evil_person who knows that a service is cloud-based to manipulate service usage billing in orders of magnitude that could be disguised easily as legitimate use of the service but drive costs to unmanageable levels. 

If you take Amazon's AWS usage-based pricing model (check out the cost calculator here,) one might envision that instead of worrying about a lack of resources, the elasticity of the cloud could actually provide a surplus of compute, network and storage utility that could be just as bad as a deficit.

Instead of worrying about Distributed Denial of Service (DDos) attacks from botnets and the like, imagine having to worry about delicately balancing forecasted need with capabilities like Cloudbursting to deal with a botnet designed to make seemingly legitimate requests for service to generate an economic denial of sustainability (EDoS) — where the dyamicism of the infrastructure allows scaling of service beyond the economic means of the vendor to pay their cloud-based service bills.

Imagine the shock of realizing that the outsourcing to the cloud to reduce CapEx and move to an OpEx model just meant that while availability, confidentiality and integrity of your service and assets are solid, your sustainability and survivability are threatened.

I know there exists the ability to control instance sprawl and constrain costs, but imagine if this is managed improperly or inexactly because we can't distinguish between legitimate and targeted resource consumption from an "attack."

If you're in the business of ensuring availability of service for large events on the web for a timed event, are you really going to limit service when you think it might drive revenues?

I think this is where service governors will have to get much more
intelligent regarding how services are being consumed and how that
affects the transaction supply chain and embed logic that takes into
account financial risk when scaling to ensure EDoS doesn't kill you. 

I can't say that I haven't had similar concerns when dealing with scalability and capacity planning in hosted or dedicated co-location environments, but generally storage and compute were not billable service options I had to worry about, only network bandwidth.

Back to the mashed potatoes.

/Hoff

James Urquhart posted a summary a week or so ago of what he described as the "Big 4" players in Cloud Computing.  It was a slightly humorous pass at describing their approaches and offerings:

James provided quite a bit more (serious) detail in the text below his table which I present to you here, tarted up with a column I've added and James left off titled "Security." 

It's written in the same spirit as James' original, so feel free to take this with an equally well-provisioned grain of NaCl.  I'll be adding my own perfunctory comments with a little more detail shortly: The point here is that the quantification of what "security" means in the cloud is as abstracted and varied as the platforms that provide the service.  We're essentially being asked to take for granted and trust that the underlying mechanicals are sound and secure while not knowing where or what they are.

We don't do that with our physically-tethered operating systems today, so why should we do so with virtualization platform hypervisors and the infrastructure "data center operating systems" of the cloud?  The transparency provided by dedicated infrastructure is being obscured by virtualization and the fog of the cloud.  It's a squeezing the balloon problem.

And so far as the argument goes toward suggesting that this is no different than what we deal with n terms of SaaS today, the difference between what we might define as legacy SaaS and "cloud" is that generally it's someone elses' apps and your data in the former (ye olde ASP model.) 

In the case of the "cloud," it could be a mixture of applications and data, some of which you own, some you don't and some you're simply not even aware of, perhaps running in part on your infrastructure and someone elses'.

It should be noted also that not all cloud providers (excluding those above) even own and operate the platforms they provide you service on…they, in turn, could be utilizing shared infrastructure to provide you service, so cross-pollination of service provisioning could affect portability, reliability and security.

That is why the Big4 above stand up their own multi-billion dollar data centers; they keep the architecture proprietary so you don't have to; lots of little clouds everywhere.

/Hoff

P.S. If you're involved with platform security from any of the providers above, do contact me because I'm going to be expounding upon the security "layers" of each of these providers in as much detail as I have here shortly.  I'd suggest you might be interested in assuring it's as complete and accurate as possible 😉

"Cloud Providers Are Better At Securing Your Data Than You Are…"

To some, this is a contentious point while to others it seems entirely logical.

I must tell you that I've witnessed this very assertion as it has been raised more times in the last few days than I can count.

Before I get into any more juicy bits regarding this topic, I wonder if you wouldn't mind popping over and reading a blog post I wrote in August, 2007 titled "On-Demand SaaS Vendors Able to Secure Assets Better than Customers?" 

Come to think of it, you can read the follow-on post to that one which clearly indicated my point when Salesforce.com and Monster.com (you know, those so-called "Cloud" providers) were breached.

Forgot about those breaches, did you?  Oh, that must have been because they were SaaS providers and not Cloud providers at the time.  Gotcha.

As you read these posts, first do so within the context of what we've come to know as software as a service (SaaS.)  Then kindly re-read it and substitute 'SaaS' with 'Cloud,' won't you?

Thanks.

I have more, but I'll wait till you're done.

/Hoff

I haven't been whipped into this much of a frenzy since Hormel changed the labels on the SPAM cans in Hawaii.

PDP (of gnucitizen fame) masterfully stitched together a collection of mixed metaphors, generalizations, reductions to the ridiculous and pejoratives to produce his opus magnum on cloud computing (in)security titled "The Cloud Is Not That Insecure."

Oh.

Since I have spent the better part of my security career building large "cloud-like" services and the products that help, at a minimum, to secure them, I feel at least slightly qualified to dispute many of his points, the bulk of which are really focused on purely technology-driven mechanical analogies and platforms rather than items such as the operational, trust, political, jurisdictional, regulatory, organizational and economical issues that really go toward the "security" (or lack thereof) of "cloud-based" service.

Speaking of which, PDP's definition of the cloud is about as abstract as you can get:

Well, I'm glad we cleared that up.

At any rate, it's a seriously humorous read that would have me taking apart many of his contradictory assumptions and assertions were it not for the fact that I have actual work to do.  So, in the issue of time, I'll offer up his conclusion and you can go back and read the rest:

So, there you have it.  Those of you who "know what you are doing" are otay and thanks to security by obscurity due to a lack of trust, cloud computing is secure.  That's not confusing at all…

This probably won't end well, but…

Sigh.

/Hoff