Rational Survivability

Previously I wrote a post titled “Oh Great Security Spirit In the Cloud: Have You Seen My WAF, IPS, IDS, Firewall…” in which I described the challenges for enterprises moving applications and services to the Cloud while trying to ensure parity in compensating controls, some of which are either not available or suffer from the “virtual appliance” conundrum (see the Four Horsemen presentation on issues surrounding virtual appliances.)

Yesterday I had a lively discussion with Lori MacVittie about the notion of what she described as “edge” service placement of network-based WebApp firewalls in Cloud deployments.  I was curious about the notion of where the “edge” is in Cloud, but assuming it’s at the provider’s connection to the Internet as was suggested by Lori, this brought up the arguments in the post
above: how does one roll out compensating controls in Cloud?

The level of difficulty and need to integrate controls (or any “infrastructure” enhancement) definitely depends upon the Cloud delivery model (SaaS, PaaS, and IaaS) chosen and the business problem trying to be solved; SaaS offers the least amount of extensibility from the perspective of deploying controls (you don’t generally have any access to do so) whilst IaaS allows a lot of freedom at the guest level.  PaaS is somewhere in the middle.  None of the models are especially friendly to integrating network-based controls not otherwise supplied by the provider due to what should be pretty obvious reasons — the network is abstracted.

So here’s the rub, if MSSP’s/ISP’s/ASP’s-cum-Cloud operators want to woo mature enterprise customers to use their services, they are leaving money on the table and not fulfilling customer needs by failing to roll out complimentary security capabilities which lessen the compliance and security burdens of their prospective customers.

While many provide commoditized solutions such as anti-spam and anti-virus capabilities, more complex (but profoundly important) security services such as DLP (data loss/leakage prevention,) WAF, Intrusion Detection and Prevention (IDP,) XML Security, Application Delivery Controllers, VPN’s, etc. should also be considered for roadmaps by these suppliers.

Think about it, if the chief concern in Cloud environments is security around multi-tenancy and isolation, giving customers more comfort besides “trust us” has to be a good thing.  If I knew where and by whom my data is being accessed or used, I would feel more comfortable.

Yes, it’s difficult to do properly and in many cases means the Cloud provider has to make a substantial investment in delivery platforms and management/support integration to get there.  This is why niche players who target specific verticals (especially those heavily regulated) will ultimately have the upper hand in some of these scenarios – it’s not socialist security where “good enough” is spread around evenly.  Services like these need to be configurable (SELF-SERVICE!) by the consumer.

An example? How about Google: where’s DLP integrated into the messaging/apps platforms?  Amazon AWS: where’s IDP integrated into the VMM for introspection?

I wrote a couple of interesting posts about this (that may show up in the automated related posts lists below):

• GooglePOPs – Cloud Computing and Clean Pipes: Told Ya So…
• Re-branding Managed Services and SaaS For Security In the Cloud…1995 Never Looked So Shiny
• Clean Pipes – Less Sewerage or More Potable Water?

My customers in the Fortune 500 complain constantly that the biggest providers they are being pressured to consider for Cloud services aren’t listening to these requests — or aren’t in a position to respond.

That’s bad for everyone.

So how about it? Are services like DLP, IDP, WAF integrated into your Cloud providers’ offerings something you’d like to see rather than having to add additional providers as brokers and add complexity and cost back into Cloud?

/Hoff

I usually keep 40-50 tabs open in my browser for review when I find things worthy of review. What usually happens is the damn thing memory leaks, implodes and I lose a bunch of good stuff.  Here’s my uber-optimized and virtualized solution to this problem.  Post ’em here:

• StorageMojo – The Cloud Quadrant
• Simon Crosby, Citrix – Whither the Venerable OS?
• TechSpot – Building the unhackable netbook
• Cisco Press – I/O Consolidation in the Data Center
• ISI/USC Viterbi School Of Engineering – Rbridge (and TRILL)
• Surendra Reddy – Virtualizaton: Solution or Problem?
• CSC – Doing Business In the Cloud: What It Means For Cost, Agility and Collaboration
• CERIAS – Virtualization Is Successful Because Operating Systems Are Weak
• Mike Fratto/NWC – A GMail Failure Is Not a Cloud Failure
• Transparent Uptime – SLA’s As An Insurance Policy? Think Again.
• PC World – Google Outages Damages Cloud Credibility
• ChannelWeb – Analysis: Attack Cripples EditDNS, Underscores DDoS Danger In Cloud
• InfoQ (George Reese) – Securing A Cloud Infrastructure
• EE|Times – Ethernet Caught In Demand/Cost Squeeze
• BigDataMatters.com – Private Cloud Data: ‘Do It Yourself With Eucaplyptus’
• ChannelWeb – Google’s Cloud ‘Not Fully Redundant,’ Company Admits
• TechInciter – Is Criticism Of Google’s Outages Unfair?
• Bits Or Pieces – Is the Enterprise Ready For the Cloud?
• Dark Reading – PCI DSS Update Could Include Virtualization Security
• Alestic – Hidden Dangers In Creating Public EBS Snapshots On EC2
• Network World – ANXeBusiness Plans Private Networking for Cloud Computing
• Information Week – Web Security In the Cloud
• Rackspace – The Cassandra Project
• CloudScaling (Randy Bias) – Cloud Standards Are Misunderstood
• Wisdom Of the Clouds (James Urquhart) – Cloud Computing and the Big Rethink: Part 1

There’s more, but this is a good list.

/Hoff

VMotion - Here's Where We Are Today

A lot has been said about the wonders of workload VM portability.

Within the construct of virtualization, and especially VMware, an awful lot of time is spent on VM Mobility but as numerous polls and direct customer engagements have shown, the majority (50% and higher) do not use VMotion.  I talked about this in a post titled “The VM Mobility Myth:”

…the capability to provide for integrated networking and virtualization coupled with governance and autonomics simply isn’t mature at this point. Most people are simply replicating existing zoned/perimertized non-virtualized network topologies in their consolidated virtualized environments and waiting for the platforms to catch up. We’re really still seeing the effects of what virtualization is doing to the classical core/distribution/access design methodology as it relates to how shackled much of this mobility is to critical components like DNS and IP addressing and layer 2 VLANs.  See Greg Ness and Lori Macvittie’s scribblings.

Furthermore, Workload distribution (Ed: today) is simply impractical for anything other than monolithic stacks because the virtualization platforms, the applications and the networks aren’t at a point where from a policy or intelligence perspective they can easily and reliably self-orchestrate.

That last point about “monolithic stacks” described what I talked about in my last post “Virtual Machines Are the Problem, Not the Solution” in which I bemoaned the bloat associated with VM’s and general purpose OS’s included within them and the fact that VMs continue to hinder the notion of being able to achieve true workload portability within the construct of how programmatically one might architect a distributed application using an SOA approach of loosely coupled services.

Combined with the VM bloat — which simply makes these “workloads” too large to practically move in real time — if one couples the annoying laws of physics and current constraints of virtualization driving the return to big, flat layer 2 network architecture — collapsing core/distribution/access designs and dissolving classical n-tier application architectures — one might argue that the proposition of VMotion really is a move backward, not forward, as it relates to true agility.

That’s a little contentious, but in discussions with customers and other Social Media venues, it’s important to think about other designs and options; the fact is that the Metastructure (as it pertains to supporting protocols/services such as DNS which are needed to support this “infrastructure 2.0”) still isn’t where it needs to be in regards to mobility and even with emerging solutions like long-distance VMotion between datacenters, we’re butting up against laws of physics (and costs of the associated bandwidth and infrastructure.)

While we do see advancements in network-driven policy stickiness with the development of elements such as distributed virtual switching, port profiles, software-based vSwitches and virtual appliances (most of which are good solutions in their own right,) this is a network-centric approach.  The policies really ought to be defined by the VM’s themselves (similar to SOA service contracts — see here) and enforced by the network, not the other way around.

Further, what isn’t talked about much is something that @joe_shonk brought up, which is that the SAN volumes/storage from which most of these virtual machines boot, upon which their data is stored and in some cases against which they are archived, don’t move, many times for the same reasons.  In many cases we’re waiting on the maturation of converged networking and advances in networked storage to deliver solutions to some of these challenges.

In the long term, the promise of mobility will be delivered by a split into three four camps which have overlapping and potentially competitive approaches depending upon who is doing the design:

1. The quasi-realtime chunking approach of VMotion via the virtualization platform [virtualization architect,]
2. Integration distribution and “mobility” at the application/OS layer [application architect,] or
3. The more traditional network-based load balancing of traffic to replicated/distributed images [network architect.]
4. Moving or redirecting pointers to large pools of storage where all the images/data(bases) live [Ed. forgot to include this from above]

Depending upon the need and capability of your application(s), virtualization/Cloud platform, and network infrastructure, you’ll likely need a mash-up of all three four.  This model really mimics the differences today in architectural approach between SaaS and IaaS models in Cloud and further suggests that folks need to take a more focused look at PaaS.

Don’t get me wrong, I think VMotion is fantastic and the options it can ultimately delivery intensely useful, but we’re hamstrung by what is really the requirement to forklift — network design, network architecture and the laws of physics.  In many cases we’re fascinated by VM Mobility, but a lot of that romanticization plays on emotion rather than utilization.

So what of it?  How do you use VM mobility today?  Do you?

/Hoff

I’m an infrastructure guy. A couple of days ago I had a lightbulb go on.  If you’re an Apps person, you’ve likely already had your share of illumination.  I’ve just never thought about things from this perspective.  Please don’t think any less of me 😉

You can bet I’m talking above my pay grade here, but bear with my ramblings for a minute and help me work through this (Update: I’m very happy to see that Surendra Reddy [@sureddy – follow him] did just that with his excellent post – cross-posted in the comments below, here. Also, check out Simon Crosby’s (Citrix CTO) post “Wither the venerable OS“)

It comes down to this:

Virtual machines (VMs) represent the symptoms of a set of legacy problems packaged up to provide a placebo effect as an answer that in some cases we have, until lately, appeared disinclined and not technologically empowered to solve.

If I had a wish, it would be that VM’s end up being the short-term gap-filler they deserve to be and ultimately become a legacy technology so we can solve some of our real architectural issues the way they ought to be solved.

That said, please don’t get me wrong, VMs have allowed us to take the first steps toward defining, compartmentalizing, and isolating some pretty nasty problems anchored on the sins of our fathers, but they don’t do a damned thing to fix them.

VMs have certainly allowed us to (literally) “think outside the box” about how we characterize “workloads” and have enabled us to begin talking about how we make them somewhat mobile, portable, interoperable, easy to describe, inventory and in some cases more secure. Cool.

There’s still a pile of crap inside ’em.

What do I mean?

There’s a bloated, parasitic resource-gobbling cancer inside every VM.  For the most part, it’s the real reason we even have mass market virtualization today.

It’s called the operating system:

If we didn’t have resource-inefficient operating systems, handicapped applications that were incestuously hooked to them, and tons of legacy networking stuff to deal with that unholy affinity, imagine the fun we could have.  Imagine how agile and flexible we could become.

But wait, isn’t server virtualization the answer to that?

Not really.  Server virtualization like that pictured in the diagram above is just the first stake we’re going to drive into the heart of the frankenmonster that is the OS.  The OS is like Cousin Eddie and his RV.

The approach we’ve taken today is that the VMM/Hypervisor abstracts the hardware from the OS.  The applications are still stuck on top of operating systems that don’t provide much in the way of any benefit given the emergence of development frameworks/languages such as J2EE, PHP, Ruby, .NET, etc. that were built around the notions of decoupled, distributed and mashable application “fabrics.”

Every ship travels with an anchor, in the case of the VM it’s the OS.

Imagine if these applications didn’t have to worry about the resource-hogging, control-freak, I/O limiting, protected mode schizophrenia and de-privileged ring spoofing of hypervisors associated with trying not conflict with or offend the OS’s sacred relationship with the hardware beneath it.

Imagine if these application constructs were instead distributed programmatically, could intercommunicate using secure protocols and didn’t have to deal with legacy problems. Imagine if the VMM/Hypervisor really was there to enable scale, isolation, security, and management.  We’d be getting rid of an entire layer.

If that crap in the middle of the sandwich makes for inefficiency, insecurity and added cost in virtualized enterprises, imagine what it does at the Infrastructure as a Service (IaaS) layer in Cloud deployments where VMs — in whatever form — are the basis for the operational models.  We have these fat packaged VMs with OS overhead and attack surfaces that really don’t need to be there.

For example, most of the pre-packaged AMIs found on AWS are bloated general purpose operating systems with some hardening applied (if at all) but there’s just all that code… sitting there…doing nothing except taking up storage, memory and compute resources.

Why do we need this?   Why don’t we at at least see more of a push towards JEOS (Just Enough OS) in the meantime?

I think most virtualization vendors today who are moving their virtualization offerings to adapt to Cloud, are asking themselves the same questions and answering them by realizing that the real win in the long term — once enterprises are done with consolidation and virtualization and hit the next “enterprise application modernization” cycle — will be  to develop and engineer applications directly around platforms which obviate the OS.

So these virtualization players are  making acquisitions to prepare them for this next wave — the real emergence of Platform as a Service (PaaS.)

Some like Microsoft with Azure are simply starting there.  Even SaaS vendors have gone down-stack and provided PaaS offerings to further allow for connectivity, integration and security in the place they think it belongs.

In the case of VMware and their acquisition of SpringSource, that piece of bloat in the middle can be seen as simply going away; whatever you call it, it’s about disintermediating the OS completely and it seems to me that the entire notion of vApps addresses this very thing.  I’m sure there are a ton of other offerings that I simply didn’t get before that are going to make me go “AHA!” now.

I’m not sure organizationally or operationally that most enterprises can get their arms around what it means to not have that OS layer in the middle in the short term, but this new platform-oriented Cloud is really interesting.  It makes those folks who may have made the conversion from server-hugger to VM-hugger and think they were done adapting, quite uncomfortable.

It makes me uncomfortable…and giddy.

All the things I know and understand about how things at the Infrastructure layer s interacts with applications and workloads at the Infostructure layer will drastically change.  The security models will change.  The solutions will change.  Even the notion of vMotion — moving VM’s around — will change.  In fact, in this model, vMotion isn’t really relevant.

Admittedly, I’ve had to call into question over the last few days just how relevant the notion of “Infrastructure 2.0” is within this model — at least how it’s described today.

Cloud v1.0 with all it’s froth and hype is going to be nothing compared to Cloud 2.0 — the revenge of SOA, web services, BPM, enterprise architecture and the developer.  Luckily for the sake of us infrastructure folks we still have time to catch up and see the light as the VMM buys us visibility and a management plane.  However, the protocols and models for how applications interact with the network are sure going to change and accelerate due to Cloud — at least they should.

Just look at how developments such as XMPP and LISP are going to play in a PaaS-centric world…

Like I said, it’s an incomplete thought and I’m not enlightened enough to frame a better discussion in written form, but I can’t wait until the next Infrastructure 2.0 Working Group to bring this up.

I have an appreciation for such a bigger piece of the conversation now.  I just need to get more edumacated.

My head ahsplodes.

Any of this crap make sense to you?

/Hoff

…and by “eat it” I mean that how you think I mean that.  I feel for these guys, they have big targets on their backs, but that’s what happens when you’re a market leader.

To wit, there are two polarized views expressed every time Google or Amazon have an outage or service interruption given that both are constantly held up as the poster children for Cloud Computing:

1. Cloud Computing isn’t ready for prime time; if Google or Amazon can go down, why/how can I trust them with my most critical assets!?
2. Google and Amazon are just service providers; service providers have issues.  This isn’t a Cloud issue, it’s just a service issue.

The truth is somewhere in the middle.

Here’s my $0.02.  You may not like it.  Refunds will be processed by mail.

If you market yourself as the shit, you can expect some back when it hits the fan:

From Hoff's Preso: Cloudifornication - Indiscriminate Information Intercourse Involving Internet Infrastructure

Stop apologizing and live up to the hype you’re helping create.

/Hoff

Back in 2008 I wrote a piece titled “Patching the Cloud” in which I highlighted the issues associated with the black box ubiquity of Cloud and what that means to patching/upgrading processes:

Your application is sitting atop an operating system and underlying infrastructure that is managed by the cloud operator.  This “datacenter OS” may not be virtualized or could actually be sitting atop a hypervisor which is integrated into the operating system (Xen, Hyper-V, KVM) or perhaps reliant upon a third party solution such as VMware.  The notion of cloud implies shared infrastructure and hosting platforms, although it does not imply virtualization.

A patch affecting any one of the infrastructure elements could cause a ripple effect on your hosted applications.  Without understanding the underlying infrastructure dependencies in this model, how does one assess risk and determine what any patch might do up or down the stack?  How does an enterprise that has no insight into the “black box” model of the cloud operator, setup a dev/test/staging environment that acceptably mimics the operating environment?

What happens when the underlying CloudOS gets patched (or needs to be) and blows your applications/VMs sky-high (in the PaaS/IaaS models?)

How does one negotiate the process for determining when and how a patch is deployed?  Where does the cloud operator draw the line?   If the cloud fabric is democratized across constituent enterprise customers, however isolated, how does a cloud provider ensure consistent distributed service?  If an application can be dynamically provisioned anywhere in the fabric, consistency of the platform is critical.

I followed this up with a practical example when Microsoft’s Azure services experienced a hiccup due to this very thing.  We see wholesale changes that can be instantiated on a whim by Cloud providers that could alter service functionality and service availability such as this one from Google (Published Google Documents to appear in Google search) — have you thought this through?

So now as we witness ISP’s starting to build Cloud service offerings from common Cloud OS platforms and espouse the portability of workloads (*ahem* VM’s) from “internal” Clouds to Cloud Providers — and potentially multiple Cloud providers — what happens when the enterprise is at v3.1 of Cloud OS, ISP A is at version 2.1a and ISP B is at v2.9? Portability is a cruel mistress.

Pair that little nugget with the fact that even “global” Cloud providers such as Amazon Web Services have not maintained parity in terms of functionality/services across their regions*. The US has long had features/functions that the european region has not.  Today, in fact, AWS announced bringing infrastructure capabilities to parity for things like elastic load balancing and auto-scale…

It’s important to understand what happens when we squeeze the balloon.

/Hoff

*corrected – I originally said “availability zones” which was in error as pointed out by Shlomo in the comments. Thanks!

A lot of fuss has been made about run-away VM sprawl in enterprises who are heavily virtualized due to the ease with which a VM can constructed and operationalized.

I’m not convinced about the reality versus the potential of VM Sprawl, meaning that I have no evidence from anyone facing this issue to date.  I wrote about this a while ago here.

As virtualization and the attendant vendors push more from enterprise virtualization to enterprise Clouds, what I’m actually more concerned with is SaaSprawl.

This scenario describes how enterprises will deal with managing what could amount to dozens of “CloudSourced” SaaS vendors as companies edge toward Cloud adoption by cherry picking applications for externalization using SaaS as the platforms, technologies and standards catch up to allow those pesky workloads that used to run internally, to do the same externally…

Outsource email, security, CRM, ERP, Legal/HR, Purchasing, Desktop apps — all from different vendors, each with different contracts, SLA’s, data integration issues, security concerns, audit constraints, regulatory compliance hiccups.

What we likely could end up with is another illustration of a “squeezing the balloon” problem; trading off CapEx for what I call OopsEx — realizing what might amount to substituting one problem for another as you trade reduced upfront (and on-going) capital investment for what amounts to on-going management, security, compliance and service-level management issues in the long term.

Thoughts?

Does anyone know of any Public Cloud Provider (or Private for that matter) that utilizes Intel’s TXT?

Specifically, does anyone know if Amazon makes use of Intel’s TXT via their Xen-derivative VMM?

Anyone care to share whether they know of any Cloud provider that PLANS to?

Thanks in advance.

Email responses welcome also [hoff @ packetfilter .com]

/Hoff

Readers of my blog will no doubt be familiar with Roland Dobbins.  He’s commented on lots of posts here and whilst we don’t always see eye-to-eye, I really respect both his intellect and his style.

So it’s fair to say that Roland is not a shy lad.  Formerly at Cisco and now at Arbor, he’s made his position (and likely his living) on dealing with a rather unpleasant issue in the highly distributed and networked InterTubes: Distributed Denial of Service (DDoS) attacks.

A recent article in ITWire titled “DDoS, the biggest threat to Cloud Computing” sums up Roland’s focus:

“According to Roland Dobbins, solutions architect for network security specialist Arbor Networks, distributed denial of service attacks are one of the must under-rated and ill-guarded against security threats to corporate IT, and in particular the biggest threat facing cloud computing.”
…
DDOS, Dobbins claims, is largely ignored in many discussions around network and cloud computing security. “Most discussions around cloud security are centred around privacy, confidentially, the separation of data from the application logic, but the security elephant in the room that very few people seem to want to talk about is DDOS. This is the number one security threat facing the cloud model,” he told last week’s Ausnog conference in Sydney.

“In cloud computing where infrastructure is shared by potentially millions of users, DDOS attacks have the potential to have much greater impact than against single tenanted architectures,” Dobbins argues. Yet, he says, “The cloud providers emerging as leaders don’t tend to talk much about their resiliency to DDOS attacks.”

Depending upon where you stand, especially if we’re talking about Public Clouds — and large Public Cloud providers such as Google, Amazon, Microsoft, etc. — you might cock your head to one side, raise an eyebrow and focus on the sentence fragment “…and in particular the biggest threat facing cloud computing.”  One of the reasons DDoS is under-appreciated is because in relative frequency — and in the stable of solutions and skill sets to deal with them — DDoS is a long tail event.

With unplanned outages afflicting almost all major Cloud providers today, the moose on the table seems to be good ol’ internal operational issues at the moment…that’s not to say it won’t become a bigger problem as the models for networked Cloud resources changes, but as the model changes, so will the defensive options in the stable.

With the decentralization of data but the mass centralization of data centers featured by these large Cloud providers, one might see how this statement could strike fear into the hearts of potential Cloud consumers everywhere and Roland is doing his best to serve us a warning — a Public (denial of) service announcement.

Sadly, at this point, however, I’m not convinced that DDoS is “the biggest threat facing Cloud Computing” and whilst providers may not “…talk much about their resiliency to DDoS attacks,” some of that may likely be due to the fact that they don’t talk much about security at all.  It also may be due to the fact that in many cases, what we can do to respond to these attacks is directly proportional to the size of your wallet.

Large network and service providers have been grappling with DDoS for years, so have large enterprises.  Folks like Roland have been on the front lines.

Cloud will certainly amplify the issues of DDoS because of how resources — even when distributed and resiliently load balanced in elastic and “perceptively infinitely scalable” ways — are ultimately organized, offered and consumed.  This is a valid point.

But if we look at the heart of most criminal elements exploiting the Internet today (and what will become Cloud,) you’ll find that the great majority want — no, *need* — victims to be available.  If they’re not, there’s no exploiting them.  DDoS is blunt force trauma — with big, messy, bloody blows that everybody notices.  That’s simply not very good for business.

At the end of the day, I think DDoS is important to think about.  I think variations of DDoS are, too.

I think that most service providers are thinking about it and investing in technology from companies such as Cisco and Arbor to deal with it, but as Roland points out, most enterprises are not — and if Cloud has its way, they shouldn’t have to:

Paradoxically, although Dobbins sees DDOS as the greatest threat to cloud computing, he also sees it as the potential solution for organisations grappling with the complexities of securing the network infrastructure.

“One answer is to get rid of all IT systems and hand them over to an organisation that specialises in these things. If the cloud providers are following best practice and have the visibility to enable them to exert control over their networks it is possible for organisation to outsource everything to them.”

For those organisations that do run their own data centres, he suggests they can avail themselves of ‘clean pipe’ services which protect against DDOS attacks According to Nick Race, head of Arbor Networks Australia, Telstra, Optus and Nextgen Networks all offer such services.

So what about you?  Moose on the table or pea under the mattress?

/Hoff

I’ve tripped over it a couple of times.

I’ve done things to it and with it that perhaps I shouldn’t have.

I’ve even rebooted it once or twice.

On Thursday, I tried — unsuccessfully — to once and for all take down the Internet.

It’s he’s just too damned resilient for his own good. 😉

One of my heroes…and an awesome person. Thank you, Vint.

You can read about the exploits of the Infrastructure 2.0 Working Group at SRI from Greg Ness’ blog here.

/Hoff