Rational Survivability

Whilst at CloudConnect I filmed some comments with Intel, RSA, Terremark and HyTrust on Intel’s Trusted Execution Technology (TXT) and its implications in the Cloud Computing space specific to “trusted cloud” and using the underlying TPM present in many of today’s compute platforms.

The 30 minute session got cut down into more consumable sound bites, but combined with the other speakers, it does a good job setting the stage for more discussions regarding this important technology.

I’ve written previously on cloud and TXT with respect to measured launch environments and work done by RSA, Intel and VMware: More On High Assurance (via TPM) Cloud Environments. Hopefully we’ll see more adoption soon.

Related articles

• Video: Intel’s Cloud Computing Vision (insidehpc.com)

I’ll be adding a material review of this document here later, but I wanted to make sure folks know this resource exists.

It’s titled the “VMware vCloud Director Security Hardening Guide”

You can download it here (PDF)

The Table of Contents appears reasonably robust…content is TBD

/Hoff

Related articles by Zemanta

• VMWare launches six new vCloud products (zdnet.com)
• How To Wield the New vShield (Edge, App & Endpoint) (rationalsurvivability.com)
• VMware’s (New) vShield: The (Almost) Bottom Line (rationalsurvivability.com)
• VMware’s Cloud Strategy: Neither Deadwood Nor Sandalwood But A Little More Than A Collection Of Driftwoods (cloudave.com)
• VMworld: VMware pushes IT-as-a-service (v3.co.uk)
• Hoff’s 5 Rules Of Cloud Security… (rationalsurvivability.com)
• The Classical DMZ Design Pattern: How To Kill Security In the Cloud (rationalsurvivability.com)
• If You Could Have One Resource For Cloud Security… (rationalsurvivability.com)

After my initial post yesterday (How To Wield the New vShield (Edge, App & Endpoint) remarking on the general sessions I sat through on vShield, I thought I’d add some additional color given my hands-on experience in the labs today.

I will reserve more extensive technical analysis of vShield Edge and App (I didn’t get to play with endpoint as there is not a lab for that) once I spend some additional quality-time with the products as they emerge.

Because people always desire for me to pop out of the cake quickly, here you go:

You should walk away from this post understanding that I think the approach holds promise within the scope of what VMware is trying to deliver. I think it can and will offer customers choice and flexibility in their security architecture and I think it addresses some serious segmentation, security and compliance gaps. It is a dramatically impactful set of solutions that is disruptive to the security and networking ecosystem. It should drive some interesting change. The proof, as they say, will be in the vPudding.

Let me first say that from VMware’s perspective I think vShield “2.0” (which logically represents many technologies and adjusted roadmaps both old and new) is clearly an important and integral part of both vSphere and vCloud Director’s future implementation strategies. It’s clear that VMware took a good, hard look at their security solution strategy and made some important and strategically-differentiated investments in this regard.

All things told, I think it’s a very good strategy for them and ultimately their customers. However, there will be some very interesting side-effects from these new features.

vShield Edge is as disruptive to the networking space (it provides L3+ networking, VPN, DHCP and NAT capabilities at the vDC edge) as it is to the security arena. When coupled with vShield App (and ultimately endpoint) you can expect VMware’s aggressive activity in retooling their offers here to cause further hastened organic development, investment, and consolidation via M&A in the security space as other vendors seek to play and complement the reabsorption of critical security capabilities back into the platform itself.

Now all of the goodness that this renewed security strategy brings also has some warts. I’ll get into some of them as I gain more hands-on experience and get some questions answered, but here’s the Cliff Note version with THREE really important points:

1. The vShield suite is the more refined/retooled/repaired approach toward what VMware promised in delivery three years ago when I wrote about it in 2007 (Opening VMM/HyperVisors to Third Parties via API’s – Goodness or the Apocalypse?) and later in 2008 (VMware’s VMsafe: The Good, the Bad, and the Bubbly…“) and from 2009, lest we forget The Cart Before the Virtual Horse: VMware’s vShield/Zones vs. VMsafe API’s…
   _
   Specifically, as the virtualization platform has matured, so has the Company’s realization that security is something they are going to have to take seriously and productize themselves as depending upon an ecosystem wasn’t working — mostly because doing so meant that the ecosystem had to uproot entire product roadmaps to deliver solutions and it was a game of “supply vs. demand chicken.”
   _
   However, much of this new capability isn’t fully baked yet, especially from the perspective of integration and usability and even feature set capabilities such as IPv6 support. Endpoint is basically the more streamlined application of APIs and libraries for anti-malware offloading so as to relieve a third party ISV from having to write fastpath drivers that sit in the kernel/VMM and disrupt their roadmaps. vShield App is the Zones solution polished to provide inter-VM firewalling capabilities.
   _
   Edge is really the new piece here and represents a new function to represent vDC perimeterized security capabilities.Many of these features are billed — quite openly — as relieving a customer from needing to use/deploy physical networking or security products. In fact, in some cases even virtual networking products such as the Cisco Nexus 1000v are not usable/supportable. This is and example of a reasonably closed, software-driven world of Cloud where the underlying infrastructure below the hypervisor doesn’t matter…until it does.
   _
2. vShield Edge and App are, in the way they are currently configured and managed, very complex and unwieldy and the performance, resiliency and scale described in some of the sessions is yet unproven and in some cases represents serious architectural deficiencies at first blush. There are some nasty single points of failure in the engineering (as described) and it’s unclear how many reference architectures for large enterprise and service provider scale Cloud use have really been thought through given some of these issues.
   _
   As an example, only being able to instantiate a single (but required) vShield App virtual appliance per ESX host brings into focus serious scale, security architecture and resilience issues. Being able to deploy numerous Edge appliances brings into focus manageability and policy sprawl concerns.There are so many knobs and levers leveraged across the stack that it’s going to be very difficult in large environments to reconcile policy spread over the three (I only interacted with two) components and that says nothing about then integrating/interoperating with third party vSwitches, physical switches, virtual and physical security appliances. If you think it was challenging before, you ain’t seen nothin’ yet.
   _
3. The current deployment methodology reignites the battle that started to rage when security teams lost visibility into the security and networking layers and the virtual administrators controlled the infrastructure from the pNIC up. This takes the gap-filler virtual security solutions from small third parties such as Altor which played nicely with vCenter but allowed the security teams to manage policy and blows that model up. Now, security enforcement is a commodity feature delivered via the virtualization platform but requires too complex a set of knowledge and expertise of the underlying virtualization platform to be rendered effective by role-driven security teams.

While I’ll cover items #1 and #2 in a follow-on post, here’s what VMware can do in the short term to remedy what I think is a huges issue going forward with item #3, usability and management.

Specifically, in the same way vCloud Director sits above vCenter and abstracts away much of the “unnecessary internals” to present a simplified service catalog of resources/services to a consumer, VMware needs to provide a dedicated security administrator’s “portal” or management plane which unites the creation, management and deployment of policy from a SECURITY perspective of the various disparate functions offered by vShield App, Edge and Endpoint. [ED: This looks as though this might be what vShield Manager will address. There were no labs covering this and no session I saw gave any details on this offering (UI or API)]

If you expect a security administrator to have the in-depth knowledge of how to administer the entire (complex) virtualization platform in order to manage security, this model will break and cause tremendous friction. A security administrator shouldn’t have access to vCenter directly or even the vCloud Director interfaces.

Since much of the capability for automation and configuration is made available via API, the notion of building a purposed security interface to do so shouldn’t be that big of a deal. Some people might say that VMware should focus on building API capabilities and allow the ecosystem to fill the void with solutions that take advantage of the interfaces. The problem is that this strategy has not produced solutions that have enjoyed traction today and it’s quite clear that VMware is interested in controlling their own destiny in terms of Edge and App while allowing the rest of the world to play with Endpoint.

I’m sure I’m missing things and that given the exposure I’ve had (without any in-depth briefings) there may be material issues associated with where the products are given their early status, but I think it important to get these thoughts out of my head so I can chart their accuracy and it gives me a good reference point to direct the product managers to when they want to scalp me for heresy.

There’s an enormous amount of detail that I want to/can get into. The last time I did that it ended up in a 150 slide presentation I delivered at Black Hat…

Allow me to reiterate what I said in the beginning:

You should walk away from this post understanding that I think the approach holds promised within the scope of what VMware is trying to deliver. I think it can and will offer customers choice and flexibility in their security architecture and I think it addresses some serious segmentation, security and compliance gaps. It is a dramatically impactful set of solutions that is disruptive to the security and networking ecosystem. It should drive some interesting change. The proof, as they say, will be in the vPudding.

…and we all love vPudding.

/Hoff

Related articles by Zemanta

• How To Wield the New vShield (Edge, App & Endpoint) (rationalsurvivability.com)
• VMWare launches six new vCloud products (zdnet.com)
• VMworld: New cloud security emphasis from VMware (v3.co.uk)
• VMware acquires Integrien, TriCipher for IT-as-a-Service Era (zdnet.com)
• Trend Micro Announces Trend Micro(TM) Deep Security 7.5: New Agentless Anti-Malware Module for VMware Environments Provides Unprecedented Security, Manageability and Performance for Dynamic Datacenters (newswire.ca)

Today at VMworld I spent my day in and out of sessions focused on the security of virtualized and cloud environments.

Many of these security sessions hinged on the release of VMware‘s new and improved suite of vShield product offerings which can be simply summarized by a deceptively simple set of descriptions:

• vShield Edge – Think perimeter firewalling for the virtual datacenter (L3 and above)
• vShield App – Think internal segmentation and zoning (L2)
• vShield Endpoint – Anti-malware service offload

The promised capabilities of these solutions offer quite a well-rounded set of capabilities from a network and security perspective but there are many interesting things to consider as one looks at the melding of the VMsafe API, vShield Zones and the nepotistic relationship enjoyed between the vCloud (nee’ VMware vCloud Director) and vSphere platforms.

There are a series of capabilities emerging which seek to solve many of the constraints associated with multi-tenancy and scale challenges of heavily virtualized enterprise and service provider virtual data center environments.  However, many of the issues associated with those I raised in the Four Horsemen of the Virtualization Security Apocalypse still stand (performance, resilience/scale, management and cost) — especially since many of these features are delivered in the form of a virtual appliance.

Many of the issues I raise above (and asked again today in session) don’t have satisfactory answers which just shows you how immature we still are in our solution portfolios.

I’ll be diving deeper into each of the components as the week proceeds (and more details around vCloud Director are made available,) but one thing is certain — there’s a very interesting amplification of the existing tug-of-war  between the security capabilities/functionality provided by the virtualization/cloud platform providers and the network/security ecosystem trying to find relevance and alignment with them.

There is going to be a wringing out of the last few smaller virtualization/Cloud security players who have not yet been consolidated via M&A or attrition (Altor Networks, Catbird, HyTrust, Reflex, etc) as the three technologies above either further highlight an identified gap or demonstrate irrelevance in the face of capabilities “built-in” (even if you have to pay for them) by VMware themselves.

Further, the uneasy tension between  the classical physical networking vendors and the virtualization/cloud platform providers is going to come to a boil, especially as it comes to configuration management, compliance, and reporting as the differentiators between simple integration at the API level of control and data plane capabilities and things like virtual firewalling (and AV, and overlay VPNs and policy zoning) begins to commoditize.

As I’ve mentioned before, it’s not where the network *is* in a virtualized environment, it’s where it *isn’t* — the definition of where the network starts and stops is getting more and more abstracted.   This in turn drives the same conversation as it relates to security.  How we’re going to define, provision, orchestrate, and govern these virtual data centers concerns me greatly as there are so many touchpoints.

Hopefully this starts to get a little more clear as more and more of the infrastructure (virtual and physical) become manageable via API such that ultimately you won’t care WHAT tool is used to manage networking/security or even HOW other than the fact that policy can be defined consistently and implemented/instantiated via API across all levels transparently, regardless of what’s powering the moving parts.

This goes back to the discussions (video) I had with Simon Crosby on who should own security in virtualized environments and why (blog).

Now all this near term confusion and mess isn’t necessarily a bad thing because it’s going to force further investment, innovation and focus on problem solving that’s simply been stalled in the absence of both technology readiness, customer appetite and compliance alignment.

More later this week. [Ed: You can find the follow-on to this post here “VMware’s (New) vShield: The (Almost) Bottom Line]

/Hoff

Related articles by Zemanta

• HyTrust Cloud Control Unveiled to Enable Accelerated Cloud Adoption (eon.businesswire.com)
• Catbird and HyTrust Team to Provide End-to-End Protection and Compliance for Virtual Infrastructure (eon.businesswire.com)
• The Classical DMZ Design Pattern: How To Kill Security In the Cloud (rationalsurvivability.com)
• The Security Hamster Sine Wave Of Pain: Public Cloud & The Return To Host-Based Protection… (rationalsurvivability.com)
• CLOUDINOMICON: Idempotent Infrastructure, Survivable Systems & Bringing Sexy Back to Information Centricity (rationalsurvivability.com)
• The Hypervisor Platform Shuffle: Pushing The Networking & Security Envelope (rationalsurvivability.com)
• Altor V4 – security in virtualized environments and the cloud (zdnet.com)

Hot on the tail of the announcement that VMware and Novell are entering into a deeper “strategic partnership” in order to deliver and support SUSE Linux Enterprise Server (SLES) for VMware vSphere environments, was an interesting blog post from Stu (@vinternals) titled “Enter the Appliance.”

Now, before we get to Stu’s post, let’s look at the language from the press release (the emphasis is mine):

VMware and Novell today announced an expansion to their strategic partnership with an original equipment manufacturer (OEM) agreement through which VMware will distribute and support the SUSE® Linux Enterprise Server operating system. Under the agreement, VMware also intends to standardize its virtual appliance-based product offerings on SUSE Linux Enterprise Server.

…

Customers who want to deploy SUSE Linux Enterprise Server for VMware® in VMware vSphere™ virtual machines will be entitled to receive a subscription to SUSE Linux Enterprise Server that includes patches and updates as part of their newly purchased qualifying VMware vSphere license and Support and Subscription. Under this agreement, VMware and its extensive network of solution provider partners will also be able to offer customers the option to purchase technical support for SUSE Linux Enterprise Server delivered directly by VMware for a seamless support experience. This expanded relationship between VMware and Novell benefits customers by reducing the cost and complexity of deploying and maintaining an enterprise operating system with VMware solutions.

As a result of this expanded collaboration, both companies intend to provide customers the ability to port their SUSE Linux-based workloads across clouds.  Such portability will deliver choice and flexibility for VMware vSphere customers and is a significant step forward in delivering the benefits of seamless cloud computing.

Several VMware products are already distributed and deployed as virtual appliances. A virtual appliance is a pre-configured virtual machine that packages an operating system and application into a self-contained unit that is easy to deploy, manage and maintain. Standardizing virtual appliance-based VMware products on SUSE Linux Enterprise Server for VMware® will further simplify the deployment and ongoing management of these solutions, shortening the path to ROI.

What I read here is that VMware virtual appliances — those VMware products packaged as virtual appliances distributed by VMware — will utilized SLES as the underlying operating system of choice. I don’t see language or the inference that other virtual appliance ISVs will be required to do so

To that point, Stu’s blog post said:

VMware will be adopting SUSE Linux Enterprise Server, SLES, as the single platform for their virtual appliances.

I’ve ranted in the past about the problem with virtual appliances. Everything from the lack of a standard Linux platform even within a single vendor (let alone amongst multiple vendors), to the additional overhead such a model of software distribution would place upon software vendors, to the security needs of the Enterprise around patch response times etc. And today, every single one of those arguments has been nullified in one fell swoop. Hallelujah, someone was listening after all!

So far, so good. Seems pretty much in-line with what VMware said.

Here’s the interesting assertion Stu makes that inspired my commentary:

If you’re a software vendor looking to adopt the virtual appliance model to distribute your wares then I have some advice for you – if you’re not using SLES for the base of your appliance, start doing so. Now. This partnership will mean doors that were previously closed to virtual appliances will now be opened, but not to any old virtual appliance – it will need to be built on an Enterprise grade distro. And SLES is most certainly that.

Chris Wolf, Stu and I had a bit of banter on Twitter regarding this announcement wherein I suggested there’s a blurring of the lines and a conflation of messaging as well as a very unique perspective that’s not being discussed.

Specifically, I don’t see where it was implied that ISV’s would be forced to adopt SLES as their OS of choice for virtual appliances.  I’m not suggesting it’s not compelling to do so for the support and distribution reasons stated above, but I suggest that the notion that “…doors that were previously closed to virtual appliances” from the perspective of support and uniformity of disto will also have and equal and opposite effect caused by a longer development lifecycle for many vendors.

Especially networking and security ISVs looking to move their products into a virtual appliance offering.

I summed up many of the issues associated with virtual security and networking appliances in my Four Horsemen of the Virtualization Security Apocalypse presentation, and given how the definition and capabilities of “the network” are (d)evolving (depending upon how you view abstraction) you might also find Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… an interesting read:

What does this mean?  It means that ultimately to ensure their own survival, virtualization and cloud providers will depend less upon virtual appliances and add more of the basic connectivity AND security capabilities into the VMMs themselves as its the only way to guarantee performance, scalability, resilience and satisfy the security requirements of customers. There will be new generations of protocols, APIs and control planes that will emerge to provide for this capability, but this will drive the same old integration battles we’re supposed to be absolved from with virtualization and Cloud.

Tell me that’s not what’s happening *right* now.

Unlike most user-facing or service-delivery applications that are not topology sensitive (that is, they simply expect to be able to speak to “the network” without knowing anything about it,) network and security ISVs do very interesting things with drivers and kernel-space code in order to deal with topology, where they sit in the stack, and how they improve performance and stability that are extremely dependent upon direct access to hardware or at the very least, customer drivers or extended/hacked kernels.

One of the reasons you see a slow trickle of network and security virtual appliances is because of these bespoke OS builds and what virtualization has done to how these services are delivered, scaled and deal with resilience.  We’ve already seen the challenge of ISVs having to re-write code to fit the VMsafe fast/slow-path driver model.

You can imagine the consternation involved if what Stu alluded to is actually required — that you must build your virtual appliances on a specific OS.  It’s going to slow down innovation and delivery of solutions if the ISV does not (for any number of valid reasons) use SLES.  This is also one of the downsides of a JEOS approach.

Stu’s warnings about compliance to SLES development notwithstanding, this puts ISVs in a delicate position — one they’ve faced before but is now exacerbated by virtualization and Cloud.  Security vendors generally minimize and harden OS stacks to fit their “application” and then tune the environment accordingly.  We’re already introducing new monocultures and uniformity in attack surfaces with hypervisors.  Are we going to do the same with the operating systems that power the virtual appliances/virtual machines that run atop them — especially those designed to protect these very systems?

Diversity is a good thing — at least when it comes to your networking and security infrastructure.  While I happen to work for a networking vendor, we all recognize that uniformity brings huge benefits as well as the potential for nasty concerns.  If you want an example, check out how a simple software error affected tens of millions of users of WordPress (WordPress and the dark side of multitenancy.) While we’re talking about a different layer in the stack, the issue is the same.

I totally grok the standardization argument for the cost control, support and manageability reasons Stu stated but I am also fearful of the extreme levels of lock-in and monoculture this approach can take.

/Hoff

Related articles by Zemanta

• VMware, Novell partner on Linux, virtualization (infoworld.com)
• Novell: The Microsoft-Taxed Fog Computing Solution (techrights.org)
• Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… (rationalsurvivability.com)
• Virtualization & Cloud Don’t Offer An *Information* Security Renaissance… (rationalsurvivability.com)
• The Four Horsemen Of the Virtualization (and Cloud) Security Apocalypse… (rationalsurvivability.com)
• Security: In the Cloud, For the Cloud & By the Cloud… (rationalsurvivability.com)
• Incomplete Thought: The Other Side Of Cloud – Where The (Wild) Infrastructure Things Are… (rationalsurvivability.com)
• Virtual Networking/Nexus 1000v Virtual Switch Blogger Roundtable/WebEx Logistics – March 2nd. (rationalsurvivability.com)
• Patching the (Hypervisor) Platform: How Do You Manage Risk? (rationalsurvivability.com)
• The Hypervisor Platform Shuffle: Pushing The Networking & Security Envelope (rationalsurvivability.com)

Back in September 2009 after presenting at the Intel Virtualization (and Cloud) Security Summit and urging Intel to lead by example by pushing the adoption and use of TPM in virtualization and cloud environments, I blogged a simple question (here) as to the following:

Does anyone know of any Public Cloud Provider (or Private for that matter) that utilizes Intel’s TXT?

Interestingly the replies were few; mostly they were along the lines of “we’re considering it,” “…it’s on our long radar,” or “…we’re unclear if there’s a valid (read: economically viable) use case.”

At this year’s RSA Security Conference, however, EMC/RSA, Intel and VMware made an announcement regarding a PoC of their “Trusted Cloud Infrastructure,” describing efforts to utilize technology across the three vendors’ portfolios to make use of the TPM:

The foundation for the new computing infrastructure is a hardware root of trust derived from Intel Trusted Execution Technology (TXT), which authenticates every step of the boot sequence, from verifying hardware configurations and initialising the BIOS to launching the hypervisor, the companies said.

Once launched, the VMware virtualisation environment collects data from both the hardware and virtual layers and feeds a continuous, raw data stream to the RSA enVision Security Information and Event Management platform. The RSA enVision is engineered to analyse events coming through the virtualisation layer to identify incidents and conditions affecting security and compliance.

The information is then contextualised within the Archer SmartSuite Framework, which is designed to present a unified, policy-based assessment of the organisation’s security and compliance posture through a central dashboard, RSA said.

It should be noted that in order to take advantage of said solution, the following components are required: a future release of RSA’s Archer GRC console, the upcoming Intel Westmere CPU and a soon-to-be-released version of VMware’s vSphere.  In other words, this isn’t available today and will require upgrades up and down the stack.

Sam Johnston today pointed me toward an announcement from Enomaly referencing the “High Assurance Edition” of ECP which laid claims of assurance using the TPM beyond the boundary of the VMM to include the guest OS and their management system:

Enomaly’s Trusted Cloud platform provides continuous security assurance by means of unique, hardware-assisted mechanisms. Enomaly ECP High Assurance Edition provides both initial and ongoing Full-Stack Integrity Verification to enable customers to receive cryptographic proof of the correct and secure operation of the cloud platform prior to running any application on the cloud.

• Full-Stack Integrity Verification provides the customer with hardware-verified proof that the cloud stack (encompassing server hardware, hypervisor, guest OS, and even ECP itself) is intact and has not been tampered with. Specifically, the customer obtains cryptographically verifiable proof that the hardware, hypervisor, etc. are identical to reference versions that have been certified and approved in advance. The customer can therefore be assured, for example, that:
• The hardware has not been modified to duplicate data to some storage medium of which the application is not aware
• No unauthorized backdoors have been inserted into the cloud managment system
• The hypervisor has not been modified (e.g. to copy memory state)
• No hostile kernel modules have been injected into the guest OS

This capability therefore enables customers to deploy applications to public clouds with confidence that the confidentiality and integrity of their data will not be compromised.

Of particular interest was Enomaly’s enticement of service providers with the following claim:

…with Enomaly’s patented security functionality, can deliver a highly secure Cloud Computing service – commanding a higher price point than commodity public cloud providers.

I’m looking forward to exploring more regarding these two example solutions as they see the light of day (and how long this will take given the need for platform-specific upgrades up and down the stack) as well as whether or not customers are actually willing to pay — and providers can command — a higher price point for what these components may offer.  You can bet certain government agencies are interested.

There are potentially numerous benefits with the use of this technology including security, compliance, assurance, audit and attestation capabilities (I hope also to incorporate more of what this might mean into the CloudAudit/A6 effort) but I’m very interested as to the implications on (change) management and policy, especially across heterogeneous environments and the extension and use of TPM’s across mobile platforms.

Of course, researchers are interested in these things too…see Rutkowska, et. al and “Attacking Intel Trusted Execution Technology” as an example.

/Hoff

Related articles by Zemanta

• Quick Question: Any Public Cloud Providers Using Intel TXT? (rationalsurvivability.com)
• EMC, Intel, VMware team to secure private clouds (sfgate.com)
• EMC, Intel, VMware team up on cloud computing security (newstatesman.com)
• RSA extends security tools to VMware (infoworld.com)
• Cloud Computing Security: What The Big Guns Have In Store (readwriteweb.com)
• Intel Launches Its Most Secure Data Center Processor (eon.businesswire.com)
• Enomaly ECP 3.1.0 Service Provider Edition Released (elasticvapor.com)

Back in October 2008, I wrote a post detailing efforts around the Xen community to create a standard security introspection API (Xen.Org Launches Community Project To Bring VM Introspection to Xen🙂

The Xen Introspection Project is a community effort within Xen.org to leverage the existing research presented above with other work not yet public to create a standard API specification and methodology for virtual machine introspection.

That blog was focused on introspection for virtualization proper but since many of the larger cloud providers utilize Xen virtualization as an underpinning of their service architecture and as an industry we’re suffering from a lack of visibility and deployable security capabilities, the relevance of VM and VMM introspection to cloud computing is quite relevant.

I thought I’d double around and see where we are.

It looks as though there’s been quite a bit of recent activity from the folks at Georgia Tech (XenAccess Project) and the University of Alaska at Fairbanks (Virtual Introspection for Xen) referenced in my previous blog.  The vCloud API proffered via the DMTF seems to also leverage (at least some of) the VMsafe API capabilities present in VMware‘s vSphere virtualization platform.

While details are, for obvious reasons sketchy, I am encouraged in speaking to representatives from a few cloud providers who are keenly interested in including these capabilities in their offerings.  Wouldn’t that be cool?

Adoption and inclusion of introspection capabilities will overcome some of the inherent security and visibility limitations we face in highly-virtualized multi-tenant environments due to networking constraints for integrating security functionality that I wrote about here.

I plan a follow-on blog in more detail once I finish some interviews.

/Hoff

Related articles by Zemanta

• Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… (rationalsurvivability.com)
• The Cloud & eHarmony’s 29 Dimensions Of Compatability… (rationalsurvivability.com)
• Incomplete Thought: Virtual Machines Are the Problem, Not the Solution… (rationalsurvivability.com)
• Variety & Darwinism In Solutions Is Innovation, In Standards It’s A War? (rationalsurvivability.com)
• Silent Lucidity: IaaS – Already A Dinosaur? The Evolution of PaaSasaurus Rex… (rationalsurvivability.com)
• Incomplete Thought: Storage In the Cloud: Winds From the ATMOS(fear) (rationalsurvivability.com)
• Redux: Patching the Cloud (rationalsurvivability.com)
• Cloud: Security Doesn’t Matter (Or, In Cloud, Nobody Can Hear You Scream) (rationalsurvivability.com)
• The Emotion of VMotion… (rationalsurvivability.com)
• Standards-Based Virtualization: Critical To The Future Of Cloud Computing (techcrunchit.com)
• VMware cloud initiative raises vendor lock-in concerns (infoworld.com)
• Where Are the Network Virtual Appliances? (gigaom.com)
• Is Virtualization Magic? (And Other Questions Your Manager May Ask) (readwriteweb.com)
• A Fluid Network is the Result of Collaboration Not Virtualization (devcentral.f5.com)
• The Road To Open Federated Clouds: Xen, VMware And More (cloudave.com)

VMotion - Here's Where We Are Today

A lot has been said about the wonders of workload VM portability.

Within the construct of virtualization, and especially VMware, an awful lot of time is spent on VM Mobility but as numerous polls and direct customer engagements have shown, the majority (50% and higher) do not use VMotion.  I talked about this in a post titled “The VM Mobility Myth:”

…the capability to provide for integrated networking and virtualization coupled with governance and autonomics simply isn’t mature at this point. Most people are simply replicating existing zoned/perimertized non-virtualized network topologies in their consolidated virtualized environments and waiting for the platforms to catch up. We’re really still seeing the effects of what virtualization is doing to the classical core/distribution/access design methodology as it relates to how shackled much of this mobility is to critical components like DNS and IP addressing and layer 2 VLANs.  See Greg Ness and Lori Macvittie’s scribblings.

Furthermore, Workload distribution (Ed: today) is simply impractical for anything other than monolithic stacks because the virtualization platforms, the applications and the networks aren’t at a point where from a policy or intelligence perspective they can easily and reliably self-orchestrate.

That last point about “monolithic stacks” described what I talked about in my last post “Virtual Machines Are the Problem, Not the Solution” in which I bemoaned the bloat associated with VM’s and general purpose OS’s included within them and the fact that VMs continue to hinder the notion of being able to achieve true workload portability within the construct of how programmatically one might architect a distributed application using an SOA approach of loosely coupled services.

Combined with the VM bloat — which simply makes these “workloads” too large to practically move in real time — if one couples the annoying laws of physics and current constraints of virtualization driving the return to big, flat layer 2 network architecture — collapsing core/distribution/access designs and dissolving classical n-tier application architectures — one might argue that the proposition of VMotion really is a move backward, not forward, as it relates to true agility.

That’s a little contentious, but in discussions with customers and other Social Media venues, it’s important to think about other designs and options; the fact is that the Metastructure (as it pertains to supporting protocols/services such as DNS which are needed to support this “infrastructure 2.0”) still isn’t where it needs to be in regards to mobility and even with emerging solutions like long-distance VMotion between datacenters, we’re butting up against laws of physics (and costs of the associated bandwidth and infrastructure.)

While we do see advancements in network-driven policy stickiness with the development of elements such as distributed virtual switching, port profiles, software-based vSwitches and virtual appliances (most of which are good solutions in their own right,) this is a network-centric approach.  The policies really ought to be defined by the VM’s themselves (similar to SOA service contracts — see here) and enforced by the network, not the other way around.

Further, what isn’t talked about much is something that @joe_shonk brought up, which is that the SAN volumes/storage from which most of these virtual machines boot, upon which their data is stored and in some cases against which they are archived, don’t move, many times for the same reasons.  In many cases we’re waiting on the maturation of converged networking and advances in networked storage to deliver solutions to some of these challenges.

In the long term, the promise of mobility will be delivered by a split into three four camps which have overlapping and potentially competitive approaches depending upon who is doing the design:

1. The quasi-realtime chunking approach of VMotion via the virtualization platform [virtualization architect,]
2. Integration distribution and “mobility” at the application/OS layer [application architect,] or
3. The more traditional network-based load balancing of traffic to replicated/distributed images [network architect.]
4. Moving or redirecting pointers to large pools of storage where all the images/data(bases) live [Ed. forgot to include this from above]

Depending upon the need and capability of your application(s), virtualization/Cloud platform, and network infrastructure, you’ll likely need a mash-up of all three four.  This model really mimics the differences today in architectural approach between SaaS and IaaS models in Cloud and further suggests that folks need to take a more focused look at PaaS.

Don’t get me wrong, I think VMotion is fantastic and the options it can ultimately delivery intensely useful, but we’re hamstrung by what is really the requirement to forklift — network design, network architecture and the laws of physics.  In many cases we’re fascinated by VM Mobility, but a lot of that romanticization plays on emotion rather than utilization.

So what of it?  How do you use VM mobility today?  Do you?

/Hoff

I’ve covered this before in more complex terms, but I thought I’d reintroduce the topic due to a very relevant discussion I just had recently (*cough cough*)

So here’s an interesting scenario in virtualized and/or Cloud environments that make use of virtual appliances to provide security capabilities*:

Since virtual appliances (VAs) are just virtual machines (VMs) what happens when a SysAdmin spins down or moves one that happens to be your shiny new firewall protecting your production VMs behind it, accidentally or maliciously?  Brings new meaning to the phrase “failing closed.”

Without getting into the vagaries of vendor specific mobility-enabled/enabling technologies, one of the issues with VMs/VAs is that there’s not really a good way of designating one as being “more important” or functionally differentiated such as “security” or “critical application” that would otherwise ensure a higher priority for service availability (read: don’t spin this down unless…) or provide a topological dependency hierarchy in virtualized network constructs.

Unlike physical environments where system administrators (servers) are segregated from access to network and security appliances, this isn’t the case in virtual environments. In Cloud environments (especially public, multi-tenant) where we are often reliant only upon virtual security capabilities since we have no option for physical alternatives, this is an interesting corner case.

We’ve talked a lot about visibility, audit and policy management in virtual environments and this is a poignant example.

/Hoff

*Despite the silly notion that the Google dudes tried to suggest I equated virtualization with Cloud as one-in-the-same, I don’t.

I was at Citrix Synergy/Virtualization Congress earlier this week and at the end of the day on Wednesday, Scott Lowe tweeted something interesting when he said:

In my mind, the biggest announcement that no one is talking about is virtual switching for XenServer. #CitrixSynergy

I had missed the announcements since I didn’t get to many of the sessions due to timing, so I sniffed around based on Scott’s hints and looked for some more meat.

I found that Chris Wolf covered the announcement nicely in his blog here but I wanted a little more detail, especially regarding approach, architecture and implementation.

Imagine my surprise when Alessandro Perilli and I sat down for a quick drink only to be joined by Simon Crosby and Ian Pratt.  Sometimes, membership has its privileges 😉

I asked Simon/Ian about the new virtual switch because I was very intrigued, and since I had direct access to the open source, it was good timing.

Now, not to be a spoil-sport, but there are details under FreiNDA that I cannot disclose, so I’ll instead riff off of Chris’ commentary wherein he outlined the need for more integrated and robust virtual networking capabilities within or adjunct to the virtualization platforms:

Cisco had to know that it was only a matter of time before competition for the Nexus 1000V started to emerge, and it appears that a virtual switch that competes with the Nexus 1000V will come right on the heels of the 1000V release. There’s no question that we’ve needed better virtual infrastructure switch management, and an overwhelming number of Burton Group clients are very interested in this technology. Client interest has generally been driven by two factors:

• Fully managed virtual switches would allow the organization’s networking group to regain control of the network infrastructure. Most network administrators have never been thrilled with having server administrators manage virtual switches.
• Managed virtual switches provide more granular insight into virtual network traffic and better integration with the organization’s existing network and security management tools

I don’t disagree with any of what Chris said, except that I do think that the word ‘compete’ is an interesting turn of phrase.

Just as the Cisco 1000v is a mostly proprietary (implementation of a) solution bound to VMware’s platform, the new Citrix/Xen/KVM virtual networking capabilities — while open sourced and free — are bound to Xen and KVM-based virtualization platforms, so it’s not really “competitive” because it’s not going to run in VMware environments. It is certainly a clear shot across the bow of VMware to address the 1000v, but there’s a tradeoff here as it comes to integration and functionality as well as the approach to what “networking” means in a virtualized construct.  More on that in a minute.

I’m going to take Chris’ next chunk out of order in order to describe the features we know about:

I’m expecting Citrix to offer more details of the open source Xen virtual switch in the near future, but in the mean time, here’s what I can tell you:

• The virtual switch will be open source and initially compatible with both Xen- and KVM-based hypervisors
• It will provide centralized network management
• It will support advanced network management features such as Netflow, SPAN, RSPAN, and ERSPAN
• It will initially be available as a plug-in to XenCenter
• It will support security features such as ACLs and 802.1x

This all sounds like good stuff.  It brings the capabilities of virtual networking and how it’s managed to “proper” levels.  If you’re wondering how this is going to happen, you *cough* might want to take a look at OpenFlow…being able to enforce policies and do things similar to the 1000v with VMware’s vSphere, DVS and the up-coming VN-Link/VN-tag is the stuff I can’t talk about — even though it’s the most interesting.  Suffice it to say there are some very interesting opportunities here that do not require proprietary networking protocols that may or may not require uplifts or upgrades of routers/switches upstream.  ’nuff said. 😉

Now the next section is interesting, but in my opinion is a bit of reach in certain sections:

For awhile I’ve held the belief that the traditional network access layer was going to move to the virtual infrastructure. A large number of physical network and security appliance vendors believe that too, and are building or currently offering products that can be deployed directly to the virtual infrastructure. So for Cisco, the Nexus 1000V was important because it a) gave its clients functionality they desperately craved, but also b) protected existing revenue streams associated with network access layer devices. Throw in an open source managed virtual switch, and it could be problematic for Cisco’s continued dominance of the network market. Sure, Cisco’s competitors can’t go at Cisco individually, but by collectively rallying around an open source managed virtual switch, they have a chance. In my opinion, it won’t be long before the Xen virtual switch can be run via software on the hypervisor and will run on firmware on SR-IOV-enabled network interfaces or converged network adapters (CNAs).

…

This is clearly a great move by Citrix. An open source virtual switch will allow a number of hardware OEMs to ship a robust virtual switch on their products, while also giving them the opportunity to add value to both their hardware devices (e.g., network adapters) and software management suites. Furthermore, an open source virtual switch that is shared by a large vendor community will enable organizations to deploy this virtual switch technology while avoiding vendor lock-in.

Firstly, I totally agree that it’s fantastic that this capability is coming to Xen/KVM platforms.  It’s a roadmap item that has been missing and was, quite honestly, going to happen one way or another.

You can expect that Microsoft will also needto respond to this some point to allow for more integrated networking and security capabilities with Hyper-V.

However, let’s compare apples to apples here.

I think it’s interesting that Chris chose to toss in the “vendor lock-in” argument as it pertains to virtual networking and virtualization for the following reasons:

• Most enterprise networking environments (from the routing & switching perspective) are usually provided by a single vendor.
• Most enterprises choose a virtualization platform from a single vendor

If you take those two things, then for an environment that has VMware and Cisco, that “lock-in” is a deliberate choice, not foisted upon them.

If an enterprise chooses to invest based upon functionality NOT available elsewhere due to a tight partnership between technology companies, it’s sort of goofy to suggest lock-in.  We call this adoption of innovation.  When you’re a competitor who is threatened because don’t have the capability you call it lock-in. ;(

This virtual switch announcement does nothing to address “lock-in” for customers who choose to run VMware with a virtual networking stack other than VMware’s or Cisco’s…see what I mean.  it doesn’t matter if the customer has Juniper switches or not in this case…until you can integrate an open source virtual switch into VMware the same way Cisco did with the 1000v (which is not trivial,) then we are where we are.

Of course the 1000v was a strategic decision by Cisco to help re-claim the access layer that was disappering into the virtualized hosts and make Cisco more relevant in a virtualized environment.  It sets the stage, as I have mentioned, for the longer term advancements of the entire Nexus and NG datacenter switching/routing products including the VN-Link/VN-Tag — with some features being proprietary and requiring Cisco hardware and others not.

I just don’t buy the argument that an open virtual switch “… could be problematic for Cisco’s continued dominance of the network market.” when the longtime availablity of open source networking products (including routers like Vyatta) haven’t made much of a dent in the enterprise against Cisco.

Customers want “open enough” and solutions that are proven and time tested.  Even the 1000v is brand new.  We haven’t even finished letting the paint dry there yet!

Now, I will say that if IBM or HP want to stick their thumb in the pie and extend their networking reach into the host by integrating this new technology with their hardware network choices, it offers a good solution — so long as you don’t mind *cough* “lock-in” from the virtualization platform provider’s perspective (since VMware is clearly excluded — see how this is a silly argument?)

The final point about “security inspection” and comparing the ability to redirect flows at a kernel/network layer to a security VA/VM/appliance  is only one small part of what VMware’s VMsafe does:

Citrix needed an answer to the Nexus 1000V and the advanced security inspection offered by VMsafe, and there’s no doubt they are on the right track with this announcement.

Certainly, it’s the first step toward better visibility and does not require API modification of the security virtual appliances/machines like VMware’s solution in it’s full-blown implementation does, but this isn’t full-blown VM introspection, either.

Moreso, it’s a way of ensuring a more direct method of gaining better visibility and control over networking in a virtualized environment.  Remember that VMsafe also includes the ability to provide interception and inspection of virtualized memory, disk, CPU execution as well as networking.  There are, as I have mentioned Xen community projects to introduce VM introspection, however.

So yes, they’re on the right track indeed and will give people pause when evaluating which virtualization and network vendor to invest in should there be a greenfield capability to do so.  If we’re dealing with environments that already have Cisco and VMware in place, not so much.

/Hoff