Rational Survivability

This is truly one of those times that we’re just going to have to hold our breath and wait and see…

Prior to VMworld, I blogged about the expected announcement by Cisco and VMware that the latter would be opening the HyperVisor to third party vendors to develop their virtual switches for ESX.

This is extremely important in the long term because security vendors today who claim to have security solutions for virtualized environments are basically doing nothing more than making virtual appliance versions of their software that run as yet another VM on a host along side critical applications.

These virtual appliances/applications are the same you might find running on their stand-alone physical appliance counterparts, and they have no access to the HyperVisor (or the vSwitch) natively.  Most of them therefore rely upon enabling promiscuous mode on vSwitches to gain visibility to inter-VM traffic which uncorks a nasty security genie of its own.  Furthermore, they impose load and latencies on the VM’s as they compete for resources with the very assets they seek to protect.

The only exception to this that I know of currently is Blue Lane who actually implements their VirtualShield product as a HyperVisor Plug-in which gives them tremendous advantages over products like Reflex and Catbird (which I will speak about all of these further in a follow-on post.)  Ed: I have been advised that this statement needs revision based upon recent developments — I will, as I mention, profile a comparison of Blue Lane, Catbird and Reflex in a follow-on post.  Sorry for the confusion.

At any rate, the specific vSwitch announcement described above was not forthcoming at the show, but a more important rumbling became obvious on the show floor after speaking with several vendors such as Cisco, Blue Lane, Catbird and Reflex; VMware was quietly beginning to provide third parties access to the HyperVisor by exposing API’s per this ZDNet article titled "VMware shares secrets in security drive":

Virtualization vendor VMware has quietly begun sharing some of
its software secrets with the IT security industry under an unannounced
plan to create better ways of securing virtual machines. 

VMware
has traditionally restricted access to its hypervisor code and, while
the vendor has made no official announcement about the API sharing
program tentatively called "Vsafe," VMware founder and chief scientist
Mendel Rosenblum said that the company has started sharing some APIs
(application program interfaces) with security vendors.

I know I should be happy about this, and I am, but now that we’re getting closer to the potential for better VM security, the implementation deserves some scrutiny.  We don’t have that yet because most of the vSafe detail is still hush-hush.

This is a double-edged sword.  While it represents a fantastic opportunity to expose functionality and provide visibility into the very guts of the VMM to allow third party software to interact with and control the HyperVisor and dependent VM/GuestOS’s, opening the Kimono represents a potentially huge new attack surface for potential malicious use.

"We would like at a high level for (VMware’s platform) to be a better
place to run," he said. "To try and realize that vision, we have been
partnering with experts in security, like the McAfees and Symantecs,
and asking them about the security issues in a virtual world."

I’m not quite sure I follow that logic.  McAfee and Symantec are just as clueless as the bulk of the security world when it comes to security issues in a virtual world.  Their answer is usually "do what you normally do and please make sure to buy a license for our software on each VM!" 

The long-term for McAfee and Symantec can’t be to continue to deploy bloatware on every VM.  Users won’t put up with the performance hit or the hit in their wallet.  They will have to re-architect to take advantage of the VMM API’s just like everyone else, but they have a more desperate timefame:

Mukil Kesavan, a VMware intern studying at the University of Rochester,
demonstrated his research into the creation of a host-based antivirus
scanning solution for virtualized servers at the conference. Such a
solution would enable people to pay for a single antivirus solution
across a box running multiple virtual servers, rather than having to
buy an antivirus solution for each virtual machine.

Licensing is going to be very critical to companies like these two very shortly as it’s a "virtual certainty" that the cost savings enjoyed by consolidating physical servers will place pressure on reducing the software licensing that goes along with it — and that includes security.

Rosenblum says that some of the traditional tools used to protect a hardware server work just as well in a virtualized environment, while others "break altogether."

"We’re trying to fix the things that break, to bring ourselves up to
the level of security where physical machines are," he said. "But we
are also looking to create new types of protection."

Rosenblum said the APIs released as part of the initiative
offer security vendors a way to check the memory of a processor, "so
they can look for viruses or signatures or other bad things."

Others allow a security vendor to check the calls an
application within a virtual machine is making, or at the packets the
machine is sending and receiving, he said.

I think Rosenblum’s statement is interesting in a couple of ways:

1. His goal, as quoted,  is to fix the things that virtualization breaks and bring security up to the level of physical servers.  Unlike every other statement from VMware spokesholes, this statement therefore suggests that virtualized environments are less secure than physical ones.  Huh.
2. I think this area of focus — when combined with the evolution of the determina acquisition — will yield excellent security gains.  Extending the monitoring and visibility into the isolated memory spaces of the virtual processors in a VM means that we may be able to counter attacks without having to depend upon solely on the virtual switches; it gives you application-level visibility without the need for another agent.

The Determina acquisition is really a red herring for VMware.  Determina’s "memory firewall" seeks to protect a system "…from buffer overflow
attacks, while still allowing the system to run at high speeds. It also
developed "hot-patching" technology–which allows servers to be patched
on the fly, while they are still running."  I’ve said before that this acquisition was an excellent move.  Let’s hope the integration goes well.

If you imagine this chunk built into the VMM, the combination of exposed VMM API’s with a lightweight VMM running in hardware (flash) embedded natively into a server less the bloated service console, it really is starting tohead down an interesting path.  This is what ESX Server 3i is designed to provide:

ESX Server 3i has considerable advantages over its predecessors
from a security standpoint. In this latest release, which will be
available in November, VMware has decoupled the hypervisor from the
service console it once shipped with. This console was based on a
version of the Red Hat Linux operating system.

As such, ESX 3i is a mere 32MB in size, rather than 2GB.

Some 50 percent of the vulnerabilities that VMware was patching in
prior versions of its software were attributable to the Red Hat piece,
not the hypervisor.

"Our hope is that those vulnerabilities will all be gone in 3i," Rosenblum said.

Given Kris Lamb’s vulnerability distribution data from last week, I can imagine that everyone hopes that these vulnerabilities will all be gone, too.  I wonder if Kris can go back and isolate how many of the vulns listed as "First Party" were attributable to the service console (the underlying RH Linux OS) accompanying the HyperVisor.  This would be good to know.  Kris? 😉

At any rate, life’s about trade-offs and security’s no different.  I think that as we see the API’s open up, so will more activity designed to start tearing at the fleshy underbelly of the VMM’s.  I wonder if we’ll see attacks specific to flash hardware when 3i comes out?

/Hoff

(P.S. Not to leave XenSource or Veridian out of the mix…I’m sure that their parent companies (Citrix & microsoft) who have quite a few combined security M&A transactions behind them are not dragging their feet on security portfolio integration, either.)