Rational Survivability

I was thrilled to be invited back to the SiliconAngle Cube at VMworld 2012 where John Furrier, Dave Vellante and I spoke in depth about security, virtualization and software defined networking (SDN)

I really like the way the chat turned out — high octane, fast pace and some great questions!

Here is the amazing full list of speakers during the event.  Check it out, ESPECIALLY Martin Casado’s talk.

As I told him, I think he is like my Obi Wan…my only hope for convincing my friends at VMware that networking and security require more attention and a real embrace of the ecosystem…

I’d love to hear your feedback on the video.

/Hoff

It’s not really *that* incomplete of a thought, but I figure I’d get it down on vPaper anyway…be forewarned, it’s massively over-simplified.

Over the last five years or so, I’ve spent my time working with enterprises who are building and deploying large scale (relative to an Enterprise’s requirements, that is) virtualized data centers and private cloud environments.

For the purpose of this discussion, I am referring to VMware-based deployments given the audience and solutions I will reference.

To this day, I’m often shocked with regard to how many of these organizations that seek to provide contextualized security for intra- and inter-VM traffic seem to position an either-or decision with respect to the use of physical or virtual security solutions.

For the sake of example, I’ll reference the architectural designs which were taken verbatim from my 2008 presentation “The Four Horsemen of the Virtualization Security Apocalypse.“

If you’ve seen/read the FHOTVA, you will recollect that there are many tradeoffs involved when considering the use of virtual security appliances and their integration with physical solutions.  Notably, an all-virtual or all-physical approach will constrain you in one form or another from the perspective of efficacy, agility, and the impact architecturally, operationally, or economically.

The topic that has a bunch of hair on it is where I see many enterprises trending: obviating virtual solutions and using physical appliances only:

…the bit that’s missing in the picture is the external physical firewall connected to that physical switch.  People are still, in this day and age, ONLY relying on horseshoeing all traffic between VMs (in the same or different VLANs) out of the physical cluster machine and to an external firewall.

Now, there are many physical firewalls that allow for virtualized contexts, zoning, etc., but that’s really dependent upon dumping trunked VLAN ports from the firewall/switches into the server and then “extending” virtual network contexts, policies, etc. upstream in an attempt to flatten the physical/virtual networks in order to force traffic through a physical firewall hop — sometimes at layer 2, sometimes at layer 3.

It’s important to realize that physical firewalls DO offer benefits over the virtual appliances in terms of functionality, performance, and some capabilities that depend on hardware acceleration, etc. but from an overall architectural positioning, they’re not sufficient, especially given the visibility and access to virtual networks that the physical firewalls often do not have if segregated.

Here’s a hint, physical-only firewall solutions alone will never scale with the agility required to service the virtualized workloads that they are designed to protect.  Further, a physical-only solution won’t satisfy the needs to dynamically provision and orchestrate security as close to the workload as possible, when the workloads move the policies will generally break, and it will most certainly add latency and ultimately hamper network designs (both physical and virtual.)

Virtual security solutions — especially those which integrate with the virtualization/cloud stack (in VMware’s case, vCenter & vCloud Director) — offer the ability to do the following:

…which is to say that there exists the capability to utilize  virtual solutions for “east-west” traffic and physical solutions for “north-south” traffic, regardless of whether these VMs are in the same or different VLAN boundaries or even across distributed virtual switches which exist across hypervisors on different physical cluster members.

For east-west traffic (and even north-south models depending upon network architecture) there’s no requirement to horseshoe traffic physically. 

It’s probably important to mention that while the next slide is out-of-date from the perspective of the advancement of VMsafe APIs, there’s not only the ability to inject a slow-path (user mode) virtual appliance between vSwitches, but also utilize a set of APIs to instantiate security policies at the hypervisor layer via a fast path kernel module/filter set…this means greater performance and the ability to scale better across physical clusters and distributed virtual switching:

Interestingly, there also exists the capability to actually integrate policies and zoning from physical firewalls and have them “flow through” to the virtual appliances to provide “micro-perimeterization” within the virtual environment, preserving policy and topology.

There are at least three choices for hypervisor management-integrated solutions on the market for these solutions today:

• VMware vShield App,
• Cisco VSG+Nexus 1000v and
• Juniper vGW

Note that the solutions above can be thought of as “layer 2” solutions — it’s a poor way of describing them, but think “inter-VM” introspection for workloads in VLAN buckets.  All three vendors above also have, or are bringing to market, complementary “layer 3” solutions that function as virtual “edge” devices and act as a multi-function “next-hop” gateway between groups of VMs/applications (nee vDC.)  For the sake of brevity, I’m omitting those here (they are incredibly important, however.)

They (layer 2 solutions) are all reasonably mature and offer various performance, efficacy and feature set capabilities. There are also different methods for plumbing the solutions and steering traffic to them…and these have huge performance and scale implications.

It’s important to recognize that the lack of thinking about virtual solutions often seem to be based largely on ignorance of need and availability of solutions.

However, other reasons surface such as cost, operational concerns and compliance issues with security teams or assessors/auditors who don’t understand virtualized environments well enough.

From an engineering and architectural perspective, however, obviating them from design consideration is a disappointing concern.

Enterprises should consider a hybrid of the two models; virtual where you can, physical where you must.

If you’ve considered virtual solutions but chose not to deploy them, can you comment on why and share your thinking with us (even if it’s for the reasons above?)

/Hoff

Related articles

• Incomplete Thought: Will the Public Cloud Create a Generation Of Network Stupid?(rationalsurvivability.com)

  ---

• Virtual Networking is more than VMs and VLAN duct tape(ioshints.info)

  ---

• Overlays: Wasting Away Again In Abstractionville… (rationalsurvivability.com)
• Virtualized security routers for cloud security (net-security.org)

I wrote a piece a while ago (in 2009) titled “Virtual Machines Are Part Of the Problem, Not the Solution…” in which I described the fact that hypervisors, virtualization and the packaging that supports them — Virtual Machines (VMs) — were actually kludges.

Specifically, VMs still contain the bloat (nee: cancer) that are operating systems and carry forward all of the issues and complexity (albeit now with more abstraction cowbell) that we already suffer.  Yes, it brings a lot of GOOD stuff, too, but tolerate the analog for a minute, m’kay.

Moreover, the move in operational models such as Cloud Computing (leveraging the virtualization theme) and the up-stack crawl from IaaS to PaaS (covered also in a blog I wrote titled: Silent Lucidity: IaaS – Already A Dinosaur?) seems to indicate a general trending toward a reduction in the number of layers in the overall compute stack.

Something I saw this morning reminded me of this and its relation to how the evolution and integration of various functions — such as virtualization and security — directly into CPUs themselves are going to dramatically disrupt how we perceive and value “virtualization” and “cloud” in the long run.

I’m not going to go into much detail because there’s a metric crapload of NDA type stuff associated with the details, but I offer you this as something you may have already thought about and the industry is gingerly crawling toward across multiple platforms.  You’ll have to divine and associate the rest:

Think “Microkernels”

…and in the immortal words of Forrest Gump “That’s all I’m gonna say ’bout that.”

/Hoff

* Ray DePena humorously quipped on Twitter that “…the flying car never materialized,” to which I retorted “Incorrect. It has just not been mass produced…” I believe this progression will — and must — materialize.

Image via Wikipedia

Every day for the last week or so after their launch, I’ve been asked left and right about whether I’d spoken to CloudPassage and what my opinion was of their offering.  In full disclosure, I spoke with them when they were in stealth almost a year ago and offered some guidance as well as the day before their launch last week.

Disappointing as it may be to some, this post isn’t really about my opinion of CloudPassage directly; it is, however, the reaffirmation of the deployment & delivery models for the security solution that CloudPassage has employed.  I’ll let you connect the dots…

Specifically, in public IaaS clouds where homogeneity of packaging, standardization of images and uniformity of configuration enables scale, security has lagged.  This is mostly due to the fact that for a variety of reasons, security itself does not scale (well.)

In an environment where the underlying platform cannot be counted upon to provide “hooks” to integrate security capabilities in at the “network” level, all that’s left is what lies inside the VM packaging:

1. Harden and protect the operating system [and thus the stuff atop it,]
2. Write secure applications and
3. Enforce strict, policy-driven information-centric security.

My last presentation, “Cloudinomicon: Idempotent Infrastructure, Building Survivable Systems and Bringing Sexy Back to Information Centricity” addressed these very points. [This one is a version I delivered at the University of Michigan Security Summit]

If we focus on the first item in that list, you’ll notice that generally to effect policy in the guest, you must have a footprint on said guest — however thin — to provide the hooks that are needed to either directly effect policy or redirect back to some engine that offloads this functionality.  There’s a bit of marketing fluff associated with using the word “agentless” in many applications of this methodology today, but at some point, the endpoint needs some sort of “agent” to play*

So that’s where we are today.  The abstraction offered by virtualized public IaaS cloud platforms is pushing us back to the guest-centric-based models of yesteryear.

This will bring challenges with scale, management, efficacy, policy convergence between physical and virtual and the overall API-driven telemetry driven by true cloud solutions.

You can read more about this in some of my other posts on the topic:

• Incomplete Thought: Cloud Security IS Host-Based…At The Moment
• The Security Hamster Sine Wave Of Pain: Public Cloud & The Return To Host-Based Protection…
• Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where…
• Dear Public Cloud Providers: Please Make Your Networking Capabilities Suck Less. Kthxbye

Finally, since I used them for eyeballs, please do take a look at CloudPassage — their first (free) offerings are based upon leveraging small footprint Linux agents and a cloud-based SaaS “grid” to provide vulnerability management, and firewall/zoning in public cloud environments.

/Hoff

* There are exceptions to this rule depending upon *what* you’re trying to do, such as anti-malware offload via a hypervisor API, but this is not generally available to date in public cloud.  This will, I hope, one day soon change.

Related articles

• Revisiting Virtualization & Cloud Stack Security – Back to the Future (Baked In Or Bolted On?) (rationalsurvivability.com)
• From the Concrete To The Hypervisor: Compliance and IaaS/PaaS Cloud – A Shared Responsibility (rationalsurvivability.com)
• CloudPassage Halo-SVM and Fireware target cloud security (zdnet.com)
• Incomplete Thought: Why Security Doesn’t Scale…Yet. (rationalsurvivability.com)
• What’s The Problem With Cloud Security? There’s Too Much Of It… (rationalsurvivability.com)

After my initial post yesterday (How To Wield the New vShield (Edge, App & Endpoint) remarking on the general sessions I sat through on vShield, I thought I’d add some additional color given my hands-on experience in the labs today.

I will reserve more extensive technical analysis of vShield Edge and App (I didn’t get to play with endpoint as there is not a lab for that) once I spend some additional quality-time with the products as they emerge.

Because people always desire for me to pop out of the cake quickly, here you go:

You should walk away from this post understanding that I think the approach holds promise within the scope of what VMware is trying to deliver. I think it can and will offer customers choice and flexibility in their security architecture and I think it addresses some serious segmentation, security and compliance gaps. It is a dramatically impactful set of solutions that is disruptive to the security and networking ecosystem. It should drive some interesting change. The proof, as they say, will be in the vPudding.

Let me first say that from VMware’s perspective I think vShield “2.0” (which logically represents many technologies and adjusted roadmaps both old and new) is clearly an important and integral part of both vSphere and vCloud Director’s future implementation strategies. It’s clear that VMware took a good, hard look at their security solution strategy and made some important and strategically-differentiated investments in this regard.

All things told, I think it’s a very good strategy for them and ultimately their customers. However, there will be some very interesting side-effects from these new features.

vShield Edge is as disruptive to the networking space (it provides L3+ networking, VPN, DHCP and NAT capabilities at the vDC edge) as it is to the security arena. When coupled with vShield App (and ultimately endpoint) you can expect VMware’s aggressive activity in retooling their offers here to cause further hastened organic development, investment, and consolidation via M&A in the security space as other vendors seek to play and complement the reabsorption of critical security capabilities back into the platform itself.

Now all of the goodness that this renewed security strategy brings also has some warts. I’ll get into some of them as I gain more hands-on experience and get some questions answered, but here’s the Cliff Note version with THREE really important points:

1. The vShield suite is the more refined/retooled/repaired approach toward what VMware promised in delivery three years ago when I wrote about it in 2007 (Opening VMM/HyperVisors to Third Parties via API’s – Goodness or the Apocalypse?) and later in 2008 (VMware’s VMsafe: The Good, the Bad, and the Bubbly…“) and from 2009, lest we forget The Cart Before the Virtual Horse: VMware’s vShield/Zones vs. VMsafe API’s…
   _
   Specifically, as the virtualization platform has matured, so has the Company’s realization that security is something they are going to have to take seriously and productize themselves as depending upon an ecosystem wasn’t working — mostly because doing so meant that the ecosystem had to uproot entire product roadmaps to deliver solutions and it was a game of “supply vs. demand chicken.”
   _
   However, much of this new capability isn’t fully baked yet, especially from the perspective of integration and usability and even feature set capabilities such as IPv6 support. Endpoint is basically the more streamlined application of APIs and libraries for anti-malware offloading so as to relieve a third party ISV from having to write fastpath drivers that sit in the kernel/VMM and disrupt their roadmaps. vShield App is the Zones solution polished to provide inter-VM firewalling capabilities.
   _
   Edge is really the new piece here and represents a new function to represent vDC perimeterized security capabilities.Many of these features are billed — quite openly — as relieving a customer from needing to use/deploy physical networking or security products. In fact, in some cases even virtual networking products such as the Cisco Nexus 1000v are not usable/supportable. This is and example of a reasonably closed, software-driven world of Cloud where the underlying infrastructure below the hypervisor doesn’t matter…until it does.
   _
2. vShield Edge and App are, in the way they are currently configured and managed, very complex and unwieldy and the performance, resiliency and scale described in some of the sessions is yet unproven and in some cases represents serious architectural deficiencies at first blush. There are some nasty single points of failure in the engineering (as described) and it’s unclear how many reference architectures for large enterprise and service provider scale Cloud use have really been thought through given some of these issues.
   _
   As an example, only being able to instantiate a single (but required) vShield App virtual appliance per ESX host brings into focus serious scale, security architecture and resilience issues. Being able to deploy numerous Edge appliances brings into focus manageability and policy sprawl concerns.There are so many knobs and levers leveraged across the stack that it’s going to be very difficult in large environments to reconcile policy spread over the three (I only interacted with two) components and that says nothing about then integrating/interoperating with third party vSwitches, physical switches, virtual and physical security appliances. If you think it was challenging before, you ain’t seen nothin’ yet.
   _
3. The current deployment methodology reignites the battle that started to rage when security teams lost visibility into the security and networking layers and the virtual administrators controlled the infrastructure from the pNIC up. This takes the gap-filler virtual security solutions from small third parties such as Altor which played nicely with vCenter but allowed the security teams to manage policy and blows that model up. Now, security enforcement is a commodity feature delivered via the virtualization platform but requires too complex a set of knowledge and expertise of the underlying virtualization platform to be rendered effective by role-driven security teams.

While I’ll cover items #1 and #2 in a follow-on post, here’s what VMware can do in the short term to remedy what I think is a huges issue going forward with item #3, usability and management.

Specifically, in the same way vCloud Director sits above vCenter and abstracts away much of the “unnecessary internals” to present a simplified service catalog of resources/services to a consumer, VMware needs to provide a dedicated security administrator’s “portal” or management plane which unites the creation, management and deployment of policy from a SECURITY perspective of the various disparate functions offered by vShield App, Edge and Endpoint. [ED: This looks as though this might be what vShield Manager will address. There were no labs covering this and no session I saw gave any details on this offering (UI or API)]

If you expect a security administrator to have the in-depth knowledge of how to administer the entire (complex) virtualization platform in order to manage security, this model will break and cause tremendous friction. A security administrator shouldn’t have access to vCenter directly or even the vCloud Director interfaces.

Since much of the capability for automation and configuration is made available via API, the notion of building a purposed security interface to do so shouldn’t be that big of a deal. Some people might say that VMware should focus on building API capabilities and allow the ecosystem to fill the void with solutions that take advantage of the interfaces. The problem is that this strategy has not produced solutions that have enjoyed traction today and it’s quite clear that VMware is interested in controlling their own destiny in terms of Edge and App while allowing the rest of the world to play with Endpoint.

I’m sure I’m missing things and that given the exposure I’ve had (without any in-depth briefings) there may be material issues associated with where the products are given their early status, but I think it important to get these thoughts out of my head so I can chart their accuracy and it gives me a good reference point to direct the product managers to when they want to scalp me for heresy.

There’s an enormous amount of detail that I want to/can get into. The last time I did that it ended up in a 150 slide presentation I delivered at Black Hat…

Allow me to reiterate what I said in the beginning:

You should walk away from this post understanding that I think the approach holds promised within the scope of what VMware is trying to deliver. I think it can and will offer customers choice and flexibility in their security architecture and I think it addresses some serious segmentation, security and compliance gaps. It is a dramatically impactful set of solutions that is disruptive to the security and networking ecosystem. It should drive some interesting change. The proof, as they say, will be in the vPudding.

…and we all love vPudding.

/Hoff

Related articles by Zemanta

• How To Wield the New vShield (Edge, App & Endpoint) (rationalsurvivability.com)
• VMWare launches six new vCloud products (zdnet.com)
• VMworld: New cloud security emphasis from VMware (v3.co.uk)
• VMware acquires Integrien, TriCipher for IT-as-a-Service Era (zdnet.com)
• Trend Micro Announces Trend Micro(TM) Deep Security 7.5: New Agentless Anti-Malware Module for VMware Environments Provides Unprecedented Security, Manageability and Performance for Dynamic Datacenters (newswire.ca)

Hot on the tail of the announcement that VMware and Novell are entering into a deeper “strategic partnership” in order to deliver and support SUSE Linux Enterprise Server (SLES) for VMware vSphere environments, was an interesting blog post from Stu (@vinternals) titled “Enter the Appliance.”

Now, before we get to Stu’s post, let’s look at the language from the press release (the emphasis is mine):

VMware and Novell today announced an expansion to their strategic partnership with an original equipment manufacturer (OEM) agreement through which VMware will distribute and support the SUSE® Linux Enterprise Server operating system. Under the agreement, VMware also intends to standardize its virtual appliance-based product offerings on SUSE Linux Enterprise Server.

…

Customers who want to deploy SUSE Linux Enterprise Server for VMware® in VMware vSphere™ virtual machines will be entitled to receive a subscription to SUSE Linux Enterprise Server that includes patches and updates as part of their newly purchased qualifying VMware vSphere license and Support and Subscription. Under this agreement, VMware and its extensive network of solution provider partners will also be able to offer customers the option to purchase technical support for SUSE Linux Enterprise Server delivered directly by VMware for a seamless support experience. This expanded relationship between VMware and Novell benefits customers by reducing the cost and complexity of deploying and maintaining an enterprise operating system with VMware solutions.

As a result of this expanded collaboration, both companies intend to provide customers the ability to port their SUSE Linux-based workloads across clouds.  Such portability will deliver choice and flexibility for VMware vSphere customers and is a significant step forward in delivering the benefits of seamless cloud computing.

Several VMware products are already distributed and deployed as virtual appliances. A virtual appliance is a pre-configured virtual machine that packages an operating system and application into a self-contained unit that is easy to deploy, manage and maintain. Standardizing virtual appliance-based VMware products on SUSE Linux Enterprise Server for VMware® will further simplify the deployment and ongoing management of these solutions, shortening the path to ROI.

What I read here is that VMware virtual appliances — those VMware products packaged as virtual appliances distributed by VMware — will utilized SLES as the underlying operating system of choice. I don’t see language or the inference that other virtual appliance ISVs will be required to do so

To that point, Stu’s blog post said:

VMware will be adopting SUSE Linux Enterprise Server, SLES, as the single platform for their virtual appliances.

I’ve ranted in the past about the problem with virtual appliances. Everything from the lack of a standard Linux platform even within a single vendor (let alone amongst multiple vendors), to the additional overhead such a model of software distribution would place upon software vendors, to the security needs of the Enterprise around patch response times etc. And today, every single one of those arguments has been nullified in one fell swoop. Hallelujah, someone was listening after all!

So far, so good. Seems pretty much in-line with what VMware said.

Here’s the interesting assertion Stu makes that inspired my commentary:

If you’re a software vendor looking to adopt the virtual appliance model to distribute your wares then I have some advice for you – if you’re not using SLES for the base of your appliance, start doing so. Now. This partnership will mean doors that were previously closed to virtual appliances will now be opened, but not to any old virtual appliance – it will need to be built on an Enterprise grade distro. And SLES is most certainly that.

Chris Wolf, Stu and I had a bit of banter on Twitter regarding this announcement wherein I suggested there’s a blurring of the lines and a conflation of messaging as well as a very unique perspective that’s not being discussed.

Specifically, I don’t see where it was implied that ISV’s would be forced to adopt SLES as their OS of choice for virtual appliances.  I’m not suggesting it’s not compelling to do so for the support and distribution reasons stated above, but I suggest that the notion that “…doors that were previously closed to virtual appliances” from the perspective of support and uniformity of disto will also have and equal and opposite effect caused by a longer development lifecycle for many vendors.

Especially networking and security ISVs looking to move their products into a virtual appliance offering.

I summed up many of the issues associated with virtual security and networking appliances in my Four Horsemen of the Virtualization Security Apocalypse presentation, and given how the definition and capabilities of “the network” are (d)evolving (depending upon how you view abstraction) you might also find Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… an interesting read:

What does this mean?  It means that ultimately to ensure their own survival, virtualization and cloud providers will depend less upon virtual appliances and add more of the basic connectivity AND security capabilities into the VMMs themselves as its the only way to guarantee performance, scalability, resilience and satisfy the security requirements of customers. There will be new generations of protocols, APIs and control planes that will emerge to provide for this capability, but this will drive the same old integration battles we’re supposed to be absolved from with virtualization and Cloud.

Tell me that’s not what’s happening *right* now.

Unlike most user-facing or service-delivery applications that are not topology sensitive (that is, they simply expect to be able to speak to “the network” without knowing anything about it,) network and security ISVs do very interesting things with drivers and kernel-space code in order to deal with topology, where they sit in the stack, and how they improve performance and stability that are extremely dependent upon direct access to hardware or at the very least, customer drivers or extended/hacked kernels.

One of the reasons you see a slow trickle of network and security virtual appliances is because of these bespoke OS builds and what virtualization has done to how these services are delivered, scaled and deal with resilience.  We’ve already seen the challenge of ISVs having to re-write code to fit the VMsafe fast/slow-path driver model.

You can imagine the consternation involved if what Stu alluded to is actually required — that you must build your virtual appliances on a specific OS.  It’s going to slow down innovation and delivery of solutions if the ISV does not (for any number of valid reasons) use SLES.  This is also one of the downsides of a JEOS approach.

Stu’s warnings about compliance to SLES development notwithstanding, this puts ISVs in a delicate position — one they’ve faced before but is now exacerbated by virtualization and Cloud.  Security vendors generally minimize and harden OS stacks to fit their “application” and then tune the environment accordingly.  We’re already introducing new monocultures and uniformity in attack surfaces with hypervisors.  Are we going to do the same with the operating systems that power the virtual appliances/virtual machines that run atop them — especially those designed to protect these very systems?

Diversity is a good thing — at least when it comes to your networking and security infrastructure.  While I happen to work for a networking vendor, we all recognize that uniformity brings huge benefits as well as the potential for nasty concerns.  If you want an example, check out how a simple software error affected tens of millions of users of WordPress (WordPress and the dark side of multitenancy.) While we’re talking about a different layer in the stack, the issue is the same.

I totally grok the standardization argument for the cost control, support and manageability reasons Stu stated but I am also fearful of the extreme levels of lock-in and monoculture this approach can take.

/Hoff

Related articles by Zemanta

• VMware, Novell partner on Linux, virtualization (infoworld.com)
• Novell: The Microsoft-Taxed Fog Computing Solution (techrights.org)
• Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… (rationalsurvivability.com)
• Virtualization & Cloud Don’t Offer An *Information* Security Renaissance… (rationalsurvivability.com)
• The Four Horsemen Of the Virtualization (and Cloud) Security Apocalypse… (rationalsurvivability.com)
• Security: In the Cloud, For the Cloud & By the Cloud… (rationalsurvivability.com)
• Incomplete Thought: The Other Side Of Cloud – Where The (Wild) Infrastructure Things Are… (rationalsurvivability.com)
• Virtual Networking/Nexus 1000v Virtual Switch Blogger Roundtable/WebEx Logistics – March 2nd. (rationalsurvivability.com)
• Patching the (Hypervisor) Platform: How Do You Manage Risk? (rationalsurvivability.com)
• The Hypervisor Platform Shuffle: Pushing The Networking & Security Envelope (rationalsurvivability.com)

This is bound to be an unpopular viewpoint.  I’ve struggled with how to write it because I want to inspire discussion not a religious battle.  It has been hard to keep it an incomplete thought. I’m not sure I have succeeded 😉

I’d like you to understand that I come at this from the perspective of someone who talks to providers of service (Cloud and otherwise) and large enterprises every day.  Take that with a grain of whatever you enjoy ingesting.  I have also read some really interesting viewpoints contrary to mine, many of which I find really fascinating, just not subscribed to my current interpretation of reality.

Here’s the deal…

While our attention has turned to the wonders of Cloud Computing — specifically the elastic, abstracted and agile delivery of applications and the content they traffic in — an interesting thing occurs to me related to the relevancy of networking in a cloudy world:

All this talk of how Cloud Computing commoditizes “infrastructure” and challenges the need for big iron solutions, really speaks to compute, perhaps even storage, but doesn’t hold true for networking.

The evolution of these elements run on different curves.

Networking ultimately is responsible for carting bits in and out of compute/storage stacks.  This need continues to reliably intensify (beyond linear) as compute scale and densities increase.  You’re not going to be able to satisfy that need by trying to play packet ping-pong and implement networking in software only on the same devices your apps and content execute on.

As (public) Cloud providers focus on scale/elasticity as their primary disruptive capability in the compute realm, there is an underlying assumption that the networking that powers it is magically and equally as scaleable and that you can just replicate everything you do in big iron networking and security hardware and replace it one-for-one with software in the compute stacks.

The problem is that it isn’t and you can’t.

Cloud providers are already hamstrung by how they can offer rich networking and security options in their platforms given architectural decisions they made at launch – usually the pieces of architecture that provide for I/O and networking (such as the hypervisor in IaaS offerings.)  There is very real pain and strain occurring in these networks.  In Cloud IaaS solutions, the very underpinnings of the network will be the differentiation between competitors.  It already is today.

See Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… or Incomplete Thought: The Cloud Software vs. Hardware Value Battle & Why AWS Is Really A Grid… or Big Iron Is Dead…Long Live Big Iron… and I Love the Smell Of Big Iron In the Morning.

With the enormous I/O requirements of virtualized infrastructure, the massive bandwidth requirements that rich applications, video and mobility are starting to place on connectivity, Cloud providers, ISPs, telcos, last mile operators, and enterprises are pleading for multi-terabit switching fabrics in their datacenters to deal with load *today.*

I was reminded of this today, once again, by the announcement of a 322 Terabit per second switch.  Some people shrugged. Generally these are people who outwardly do not market that they are concerned with moving enormous amounts of data and abstract away much of the connectivity that is masked by what a credit card and web browser provide.  Those that didn’t shrug are those providers who target a different kind of consumer of service.

Abstraction has become a distraction.

Raw networking horsepower, especially for those who need to move huge amounts of data between all those hyper-connected cores running hundreds of thousands of VM’s or processes, still know it as a huge need.

Before you simply think I’m being a shill because I work for networking vendor (and the one that just announced that big switch referenced above,) please check out the relevant writings on this viewpoint which I have held for years which is that we need *both* hardware and software based networking to scale efficiently and the latter simply won’t replace the former.

Virtualization and Cloud exacerbate the network-centric issues we’ve had for years.

I look forward to the pointers to the sustainable, supportable and scaleable 322 Tb/s software-based networking solutions I can download and implement today as a virtual appliance.

/Hoff

Related articles by Zemanta

• Does the Cloud Need a Specialized Chip? (gigaom.com)
• Dear Public Cloud Providers: Please Make Your Networking Capabilities Suck Less. Kthxbye (rationalsurvivability.com)
• Can We Secure Cloud Computing? Can We Afford Not To? (rationalsurvivability.com)
• NESSessary Question: Will Virtualization Undermine Network Equipment Vendors? (rationalsurvivability.com)
• Cloud: Over Subscription vs. Over Capacity – Two Different Things (rationalsurvivability.com)
• Cloud Computing Security: (Orchestral) Maneuvers In the Dark? (rationalsurvivability.com)
• From the X-Files – The Cloud in Context: Evolution from Gadgetry to Popular Culture (rationalsurvivability.com)

It’s very interesting to see that now that infrastructure-as-a-service (IaaS) players like Amazon Web Services are clawing their way “up the stack” and adding more platform-as-a-service (PaaS) capabilities, that Microsoft is going “down stack” and providing IaaS capabilities by way of adding RDP and VM capabilities to Azure.

From Carl Brooks’ (@eekygeeky) article today:

Microsoft is expected to add support for Remote Desktops and virtual machines (VMs) to Windows Azure by the end of March, and the company also says that prices for Azure, now a baseline $0.12 per hour, will be subject to change every so often.

Prashant Ketkar, marketing director for Azure, said that the service would be adding Remote Desktop capabilities as soon as possible, as well as the ability to load and run virtual machine images directly on the platform. Ketkar did not give a date for the new features, but said they were the two most requested items.
…

This move begins a definite trend away from the original concept for Azure in design and execution. It was originally thought of as a programming platform only: developers would write code directly into Azure, creating applications without even being aware of the underlying operating system or virtual instances. It will now become much closer in spirit to Amazon Web Services, where users control their machines directly. Microsoft still expects Azure customers to code for the platform and not always want hands on control, but it is bowing to pressure to cede control to users at deeper and deeper levels.

One major reason for the shift is that there are vast arrays of legacy Windows applications users expect to be able to run on a Windows platform, and Microsoft doesn’t want to lose potential customers because they can’t run applications they’ve already invested in on Azure. While some users will want to start fresh, most see cloud as a way to extend what they have, not discard it.

This sets the path to allow those enterprise customers running HyperV internally to take those VMs and run them on (or in conjunction with) Azure.

Besides the obvious competition with AWS in the public cloud space, there’s also a private cloud element. As it stands now, one of the primary differentiators for VMware from the private-to-public cloud migration/portability/interoperability perspective is the concept that if you run vSphere in your enterprise, you can take the same VMs without modification and move them to a service provider who runs vCloud (based on vSphere.)

This is a very interesting and smart move by Microsoft.

/Hoff

Related articles by Zemanta

• Silent Lucidity: IaaS – Already A Dinosaur? The Evolution of PaaSasaurus Rex… (rationalsurvivability.com)
• Incomplete Thought: Virtual Machines Are the Problem, Not the Solution… (rationalsurvivability.com)
• Just A Reflective Bookmark: Microsoft’s Azure…The Dark Horse Emergeth… (rationalsurvivability.com)
• Cloud Pricing War Begins (cloudave.com)
• Windows Azure starts bringin’ in the cash (seattlepi.com)

Allan Leinwand from GigaOm wrote a great article asking “Where are the network virtual appliances?” This was followed up by another excellent post by Rich Miller.

Allan sets up the discussion describing how we’ve typically plumbed disparate physical appliances into our network infrastructure to provide discrete network and security capabilities such as load balancers, VPNs, SSL termination, firewalls, etc.  He then goes on to describe the stunted evolution of virtual appliances:

To be sure, some networking devices and appliances are now available in virtual form.  Switches and routers have begun to move toward virtualization with VMware’s vSwitch, Cisco’s Nexus 1000v, the open source Open vSwitch and routers and firewalls running in various VMs from the company I helped found, Vyatta.  For load balancers, Citrix has released a version of its Netscaler VPX software that runs on top of its virtual machine, XenServer; and Zeus Systems has an application traffic controller that can be deployed as a virtual appliance on Amazon EC2, Joyent and other public clouds.

Ultimately I think it prudent for discussion’s sake to separate routing, switching and load balancing (connectivity) from functions such as DLP, firewalls, and IDS/IPS (security) as lumping them together actually abstracts the problem which is that the latter is completely dependent upon the capabilities and functionality of the former.  This is what Allan almost gets to when describing his lament with the virtual appliance ecosystem today:

Yet the fundamental problem remains: Most networking appliances are still stuck in physical hardware — hardware that may or may not be deployed where the applications need them, which means those applications and their associated VMs can be left with major gaps in their infrastructure needs. Without a full-featured and stateful firewall to protect an application, it’s susceptible to various Internet attacks.  A missing load balancer that operates at layers three through seven leaves a gap in the need to distribute load between multiple application servers. Meanwhile, the lack of an SSL accelerator to offload processing may lead to performance issues and without an IDS device present, malicious activities may occur.  Without some (or all) of these networking appliances available in a virtual environment, a VM may find itself constrained, unable to take full advantage of the possible economic benefits.

I’ve written about this many, many times. In fact almost three years ago I created a presentation called  “The Four Horsemen of the Virtualization Security Apocalypse” which described in excruciating detail how network virtual appliances were a big ball of fail and would be for some time. I further suggested that much of the “best-of-breed” products would ultimately become “good enough” features in virtualization vendor’s hypervisor platforms.

Why?  Because there are some very real problems with virtualization (and Cloud) as it relates to connectivity and security:

1. Most of the virtual network appliances, especially those “ported” from the versions that usually run on dedicated physical hardware (COTS or proprietary) do not provide feature, performance, scale or high-availability parity; most are hobbled or require per-platform customization or re-engineering in order to function.
2. The resilience and high availability options from today’s off-the-shelf virtual connectivity does not pair well with the mobility and dynamism of de-coupled virtual machines; VMs are ultimately temporal and networks don’t like topological instability due to key components moving or disappearing
3. The performance and scale of virtual appliances still suffer when competing for I/O and resources on the same physical hosts as the guests they attempt to protect
4. Virtual connectivity is a generally a function of the VMM (or a loadable module/domain therein.) The architecture of the VMM has dramatic impact upon the architecture of the software designed to provide the connectivity and vice versa.
5. Security solutions are incredibly topology sensitive.  Given the scenario in #1 when a VM moves or is distributed across the pooled infrastructure, unless the security capabilities are already present on the physical host or the connectivity and security layers share a control plane (or at least can exchange telemetry,) things will simply break
6. Many virtualization (and especially cloud) platforms do not support protocols or topologies that many connectivity and security virtual appliances require to function (such as multicast for load balancing)
7. It’s very difficult to mimic the in-line path requirements in virtual networking environments that would otherwise force traffic passing through the connectivity layers (layers 2 through 7) up through various policy-driven security layers (virtual appliances)
8. There is no common methodology to express what security requirements the connectivity fabrics should ensure are available prior to allowing a VM to spool up let alone move
9. Virtualization vendors who provide solutions for the enterprise have rich networking capabilities natively as well as with third party connectivity partners, including VM and VMM introspection capabilities. As I wrote about here, mass-market Cloud providers such as Amazon Web Services or Rackspace Cloud have severely crippled networking.
10. Virtualization and cloud vendors generally force many security vs. performance tradeoffs when implementing introspection capabilities in their platforms: third party code running in the kernel, scheduler prioritization issues, I/O limitations, etc.
11. Much of the basic networking capabilities are being pushed lower into silicon (into the CPUs themselves) which makes virtual appliances even further removed from the guts that enable them
12. Physical appliances (in the enterprise) exist en-mass.  Many of them provide highly scalable solutions to the specific functions that Alan refers to.  The need exists, given the limitations I describe above, to provide for integration/interaction between them, the VMM and any virtual appliances in order to offload certain functions as well as provide coverage between the physical and the logical.

What does this mean?  It means that ultimately to ensure their own survival, virtualization and cloud providers will depend less upon virtual appliances and add more of the basic connectivity AND security capabilities into the VMMs themselves as its the only way to guarantee performance, scalability, resilience and satisfy the security requirements of customers. There will be new generations of protocols, APIs and control planes that will emerge to provide for this capability, but this will drive the same old integration battles we’re supposed to be absolved from with virtualization and Cloud.

Connectivity and security vendors will offer virtual replicas of their physical appliances in order to gain a foothold in virtualized/cloud environments in order to intercept traffic (think basic traps/ACL’s) and then interact with higher-performing physical appliance security service overlays or embedded line cards in service chassis.  This is especially true in enterprises but poses many challenges in software-only, mass-market cloud environments where what you’ll continue to get is simply basic connectivity and security with limited networking functionality.  This implies more and more security will be pushed into the guest and application logic layers to deal with this disconnect.

This is exactly where we are today with Cloud providers like Amazon Web Services: basic ingress-only filtering with a very simplistic, limited and abstracted set of both connectivity and security capability.  See “Dear Public Cloud Providers: Please Make Your Networking Capabilities Suck Less. Kthxbye”  Will they add more functionality?  Perhaps. The question is whether they can afford to in order to limit the impact that connecitivity and security variability/instability can bring to an environment.

That said, it’s certainly achievable, if you are willing and able to do so, to construct a completely software-based networking environment, but these environments require a complete approach and stack re-write with an operational expertise that will be hard to support for those who have spent the last 20 years working in a different paradigm and that’s a huge piece of this problem.

The connectivity layer — however integrated into the virtualized and cloud environments they seem — continues to limit how and what the security layers can do and will for some time, thus limiting the uptake of virtual network and security appliances.

Situation normal.

/Hoff

Related articles by Zemanta

• The Forthcoming Citrix/Xen/KVM Virtual Networking Stack…What Does This Mean to VMware/Cisco 1000v? (rationalsurvivability.com)
• Cloud Providers and Security “Edge” Services – Where’s The Beef? (rational survivability)
• Oh Great Security Spirit In the Cloud: Have You Seen My WAF, IPS, IDS, Firewall…
• Dear Public Cloud Providers: Please Make Your Networking Capabilities Suck Less. Kthxbye (rational survivability)
• The Four Horsemen Of the Virtualization Security Apocalypse (rational survivability)

• Xen packages build-your-own-cloud kit (theregister.co.uk)
• I Found the Missing Piece of the Virtualization Puzzle (devcentral.f5.com)
• CohesiveFT Rocks With VPNCubed For vCloud (cloudave.com)