Rational Survivability

Update below.

It's the night before VMworld 2008 and the Technology
Exchange/Partner day begins and I'm pawing through the stuff in my bag,
separating the "keep it" from the "toss it" schwag.

There's an innocuous little flyer stuffed in the bag on Cisco
letterhead titled "Introducing Cisco's Virtual Switch for VMware ESX." 
Fantastic.  Let's call it the 'cSwitch' 😉

A year and a month ago in August of 2007, I blogged about this very thing in a post titled: "VMware to Open Development of ESX Virtual Switches to Third Parties…Any Guess Who's First?" based on a hint from virtualization.info.

Given
that VMworld 2007 came and went without this announcement, I'm very
excited that we're actually going to get a look at what Cisco will
offer; I think this is huge news and ultimately offers some profound
game-changing (for good and bad) implications on the network and
security fronts.

I have dozens of questions like: I wonder how
much of the Nexus (7000 series)/NX-OS code cross-pollinates over (if
any) to this solution and if we'll see capabilities such as
STP/PVST+/Private VLANs, HSRP, Multicast, etc. make their way into
Cisco's vSwitch and how this virtual switch with integrate/interoperate
with the vkernel.

Further, as Ed Haletky and I unofficially bet
over drinks this evening, I wonder if it will be a direct replacement
for VMware's at-boot loadable module or it will co-exist?  I bet the
former. 😉

In addition to the "cSwitch," there are a couple of
sessions I am very, very interested in attending given my exposure to
VFrame and some Cisco engineers/architects at last year's show:

Simplify VMotion with Virtual Machine–Aware Network and Storage Services
See how network and storage services can be linked to a virtual machine so they move with VMotion events.

ESX Server in a Unified Fabric Environment
See how ESX Server works in a unified fabric environment with ESX 3.5
U2, Emulex Converged Network Adaptors, and the Cisco Nexus 5000.

VFrame: Enriching ESX Deployment with End-to-End Orchestration
Cisco’s VFrame DC 1.2 provides an easy-to-use template-based
provisioning approach for rapid, repeatable, and compliant provisioning
of ESX Servers. Through a rich set of networking and storage
orchestration capabilities, it reduces the time required to bring up
ESX clusters while providing operational scalability to manage large
clusters effectively.

See the second topic above?  Remember when I mentioned in prior posts about virtualizing applications directly within the Nexus?

Should be a very interesting couple of days.

/Hoff

Update:
So there was no direct news/mention specifically of Cisco today in any
of the distributed virtual networking (DVN) sessions — there's a lot
of messaging collisions because the re-branded 'v-everything' strategy
has things being renamed.  Hopefully we'll see/hear more from Cisco
tomorrow.

Many
of the underlying functions that will enable 3rd party virtual switches
as well as any network interface to the vkernel via API were discussed today
under the capabilities described by vNetwork (this includes the vNetwork Appliance API's and what you've known as VMsafe.)  You can see more about vNetwork here in this post.

All
I can say is that I got a lot of my suspicions confirmed, questions
answered and conclusions affirmed in today's sessions.  Some good, some
bad.  It's going to be a bumpy ride, kids.

The Four Horsemen live! 😉

The resiliency and availability of security appliances in virtual environments is a focus of my "Four Horsemen of the Virtualization Security Apocalypse" presentation which I delivered during Blackhat last week.

In my session I discussed some detailed scenarios of the architectural adjustments to infrastructure that virtualizing physical security appliances require and what that means to the overall resiliency,
reliability, performance and scalability of systems which depend upon security controls in the form of physical appliances today.

Specifically, I highlighted some of the limitations of using security virtual appliances and demonstrated how relying upon virtual security appliances can actually decrease the security posture and increase risk in our environments today.

One of my examples illustrated how it may become necessary to combine multiple security virtual appliances on a cluster separate from the VM’s they are protecting in order to achieve the availability, performance, scalability, and resiliency that we get out of dedicated in-line physical appliances today.

If you prefer a simpler example, I also presented the simpler example of a firewall virtual appliance front-ending 10 production VM’s.  I like the former because it directly contrasts the physical and completely virtual.

For the sake of illustrating a point, imagine if one were actually able to satisfy business, security and compliance requirements using this virtualized security architecture in a production environment built upon the same virtualization platform as non-security guests.*

Not to pick on my friends at VMware, but today’s license timebomb issue delivered by a VMware patch is pretty nasty and sends my hackles skyward when contemplating what it would mean were one to virtualize the security infrastructure. 

Basically, this issue caused by an update rendered certain VMs unusable after the update.  If one or more VM’s on an updated host happened to be a security appliance or security control taken down for patching or rotated for re-purposing, etc. imagine your surprise post-patch.

Remember that security applications are very topology and state-sensitive and
unlike other apps. that just care about an IP address with which to spew packets
ethereally, security appliances/applications need to bind access
policies with affinity in order to protect assets behind them.  As we
all know, when something doesn’t work, we invoke the SysAdmin prime
directive: "Blame the firewall!" 😉

Events such as this may cause some to give pause to enterprise security architects before migrating security functions to virtual appliances, especially given where we are today with what I presented in terms of high-availability options within single-cluster hosts with virtual security appliances. 

In reality, it will probably cause people to consider what virtualization means as an overall contributor to operational risk for any physical system conversion — regardless of vendor — since any VA/VM would be affected, but let’s think about this as if one had virtualized the security infrastructure.

While it’s true that for guests we have DR options and snapshotting that would make roll backs an easy affair, this shines a spotlight on the difficulties with patching the underlying virtualization platform and what that means to operational resilience.

The notion of a homogeneous virtualization platform are certainly compelling; easier administration, patching, configuration, standardization, reduced costs, etc.  However, the notion of a monoculture "operating system" has its downfalls also.  This issue clearly highlights one of them.

I’m not suggesting that there are not opportunities for virtualizing certain security functions, but as I pointed out in my talk, many of the required topologies and high-availability options present in mature physical security appliances are not available in the virtual appliance world.

Today’s issue highlights the need for very careful planning when comes to what, when and if one should
virtualize security functions.

When in doubt, refer to Hoff’s Corollary:

/Hoff

* You might want to look at what a platform like the Crossbeam X-Series can give you that "normal" virtualization security platforms cannot, as it mitigates some of the issues mentioned in my talk.

** Of relevance is my blog post from back in January titled "On Patch Tuesdays For Virtualization Platforms"

I wanted to make you aware of a "new" excellent budding resource for VMware infrastructure, VMware’s VI:Ops – Virtual Infrastructure Operations.  Steve Chambers of VMware pointed me over to the site which is growing in both content and contributors.

VI:Ops currently includes the following sections:

• Strategies and solutions using virtualization
• Building
  and managing virtual infrastructure with open, industry standards
• Securing virtual infrastructure against risk and for compliance
• Managing and operating virtual infrastructure in the enterprise
• Automate everything virtual to be agile and efficient

Check out the site and join the community!

/Hoff

I was having an interesting discussion the other evening at BeanSec with Jeanna Matthews from Clarkson University.  Jeanna is one of the authors of what I think is the best book available on Xen virtualization, Running Xen.

In between rounds of libations, the topic of Hypervisor-neutral, VM portability/interoperability between the virtualization players (see right) came up.  If I remember correctly, we were discussing the announcement from Citrix regarding Project Kensho:

Santa Clara, CA » 7/15/2008 » Citrix Systems, Inc.
(Nasdaq:CTXS), the global leader in application delivery
infrastructure, today announced “Project Kensho,” which will deliver
Open Virtual Machine Format (OVF) tools that, for the first time, allow
independent software vendors (ISVs) and enterprise IT managers to
easily create hypervisor-independent, portable enterprise application
workloads. 
These tools will allow application workloads to be imported
and run across Citrix XenServer™, Microsoft Windows Server 2008 Hyper-V™ and VMware™ ESX virtual environments. 

On the surface, this sounded like a really interesting and exciting development regarding interoperability between virtualization platforms and the VMs that run on them.  Digging deeper, however, it’s not really about virtualization at all; it’s about the delivery of applications and services — almost in spite of the virtualization layer — which is something I hinted about at the end of this post.

I am of the opinion that virtualization is simply
a means to an end, a rationalized and cost-driven stepping-stone along the path of
designing, provisioning, orchestrating, deploying, and governing a more agile, real time
infrastructure to ensure secure, resilient, cost-effective and dynamic delivery of service.

You might call the evolution of virtualization and what it’s becoming cloud computing.  You might call it utility computing.  You might call it XaaS.  What many call it today is confusing, complex, proprietary and a pain in the ass to manage.

Thus, per the press release regarding Project Kensho, the notion of packaging applications/operating environments up as tasty little hypervisor-neutral nuggets in the form of standardized
virtual appliances that can run anywhere on any platform is absolutely appealing and in the long term, quite necessary.*

However, in the short term, I am left wondering if this is a problem being "solved" for ISV’s and virtualization platform providers or for customers?  Is there a business need today for this sort of solution and is the technology available to enable it?

Given the fact that my day job and paycheck currently depends upon crafting security strategies, architecture and solutions for real time infrastructure, I’m certainly motivated to discuss this.  Mortgage payment notwithstanding, here’s a doozy of a setup:

Given where we are today with the heterogeneous complexity and nightmarish management realities of our virtualized and non-virtualized infrastructure, does this really solve relevant customer problems today or simply provide maneuvering space for virtualization platform providers who see their differentiation via the hypervisor evaporating?

While the OVF framework was initially supported by a menagerie of top-shelf players in the virtualization space, it should come as no surprise that this really represents the first round in a cage match fight to the death for who wins the application/service delivery management battle.

You can see this so clearly in the acquisition strategies of VMware, Citrix and Microsoft.

Check out the remainder of the press release.  The first half had a happy threesome of Citrix, Microsoft and VMware taking a long walk on the beach.  The second half seems to suggest that someone isn’t coming upstairs for a nightcap:

Added Value for Microsoft Hyper-V

Project Kensho will also enable customers to leverage the
interoperability benefits and compatibility between long-time partners
Citrix and Microsoft to extend the Microsoft platform.  For example,
XenServer is enhanced with CIM-based management APIs to allow any
DMTF-compliant management tool to manage XenServer, including Microsoft
System Center Virtual Machine Manager. And because the tools are based
on a standards framework, customers are ensured a rich ecosystem of
options for virtualization.  In addition, because of the open-standard
format and special licensing features in OVF, customers can seamlessly
move their current virtualized workloads to either XenServer or
Hyper-V, enabling them to distribute virtual workloads to the platform
of choice while simultaneously ensuring compliance with the underlying
licensing requirements for each virtual appliance.

Project Kensho will support the vision of the Citrix Delivery Center™
product family, helping customers transform static datacenters into
dynamic “delivery centers” for the best performance, security, cost
savings and business agility. The tools developed through Project
Kensho will be easily integrated into Citrix Workflow Studio™ based
orchestrations, for example, to provide an automated, environment for
managing the import and export of applications from any major
virtualization platform.

Did you catch the subtlety there?  (Can you smell the sarcasm?)

I’ve got some really interesting examples of how this is currently shaking out in very large enterprises.  I intend to share them with you, but first I have a question:

What relevance do hypervisor-neutral virtual appliance/machine deployments have in your three year virtualization roadmaps?  Are they a must-have or nice-to-have? Do you see deploying multiple hypervisors and needing to run these virtual appliances across any and all platforms regardless of VMM?

Of course it’s a loaded question.  Would you expect anything else?

/Hoff

* There are some really interesting trade-offs to be made when deploying virtual appliances.  This is the topic of my talk at Blackhat this year titled "The Four Horsemen of the Virtualization Apocalypse"

Edward Haletky wrote an interesting piece recently titled "CISecurity Guide to VMware Security Falls Far Short" in which he lays down some well-articulated criticisms of the first CIS benchmark for VMware.

Edward’s primary problem with the benchmark can be summarized well by this paragraph:

While the Benchmark was the first of its kind, it is nothing more than the Linux benchmark with some small changes for VMware ESX. Following these steps will increase security but it is by no means a panacea. Do not let it give you a false sense of security.

I think Edward set his expectations a little high prior to review, as I’m pretty sure the word panacea wasn’t used in the syllabus 😉

I don’t disagree with Edward that the flavor of the benchmark is very much a generic set of guidelines focused primarily on securing the underlying Linux-based service console and basic configuration for overall "system" hardening, but we need to realize a couple of things to keep the benchmark in perspective:

1. The benchmark was the first of its kind.  It’s almost 10 months old!  The second version is underway right now as a matter of fact.
2. In between when the benchmark was released and now, we’ve seen the emergence of the embedded version of VMware and much needs to change to address that.
3. The benchmark was designed to be generic and give virtual system administrators a baseline on basic security hardening, not serve as the end-all, be-all for some mythical security end-state.
4. The challenge for those of us who contributed (as I did) was that we had to keep the document vendor/tool agnostic which makes it difficult to frame solutions.
5. Lots of things have changed.

Keep in mind that this is a "level 1" benchmark whose settings/actions are as follows:

• Can be understood and performed by system administrators with any level of security knowledge and experience;
• Are unlikely to cause an interruption of service to the operating system or the applications that run on it; and
• Can be automatically monitored either by CIS Scoring Tools or by CIS Certified tools available from security software vendors. 

This isn’t about being defensive regarding the benchmark as I’ll agree that we could have done much, much more in terms of providing more meatier substance as it relates to how to better secure the ecosystem of mechanicals that a virtualized environment touches. 

However, the scope of a document that effectively addresses the security concerns across this immense landscape would be a huge undertaking.

One of the other difficulties in creating a guideline like this is the fact that those responsible for securing virtualized environments are not security professionals.  As I’ve spoken about previously, the operational realities of who is managing and securing our virtualized infrastructure is cause for concern.

Thus, when creating a guide like this, it’s best to start with the underlying basics and then branch out from there; involve the network and security teams as required.  As Edward himself wrote in this piece, "Good virtual security requires better IT teamwork," to properly secure your virtualized infrastructure, it’s going to take cooperation and expertise from many camps.    

Edward also has written a book titled "VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers."  Interestingly, I found the security sections weak for many of the same high-level reasons he listed in his review of the CIS benchmark.  Security is most definitely in the eye of the "bookholder." 😉

In the meantime, if you’re interested in some additional security/hardening guides and tools for VMware environments, check out the following:

• VMWare – Updated VI3 Security Hardening Guide (7/8/08)
• DoD/DISA – Security Technical Implementation Guide (STIG)
• NSA SNAC – VMware ESX Server 3 Configuration Guide
• Tripwire – ConfigCheck

/Hoff

Alessandro from virtualization.info and I were chatting today regarding VMware’s latest best-practices document titled "DMZ Virtualization with VMware Infrastructure." 

This is a nine page overview that does a reasonably good job of laying out many of the architectural/topological options available when thinking about taking the steps toward virtualizing what some consider the "final frontier" in the proving grounds of production-level virtualization — the (Internet-facing) DMZ.

The whitepaper was timely because I was just finishing up my presentation for Blackhat and was busy creating a similar set of high-level architectural examples to use in my presentation.  I decided to reference those in the document because they quite elegantly represent the starting points that many folks would use as a stepping off point in their virtual DMZ adventures.

…and I think it will be an adventure punctuated perhaps by evolutionary steps as documented in the options presented in the whitepaper.

As I read through the document, I had to remind myself of the fact that this was intended to be a high-level document and not designed to cover the hairy edges of network and security design. 

The whitepaper highlighted some of the reasonable trade-off’s in complexity, resiliency, management, functionality, operational expertise, and cost but given where my head and focus are today, I have to admit that it still gnawed at me from a security perspective which is still too weak for my liking.

I’ve hinted at why in my original Four Horsemen slide, and I’m going to be speaking for 75 minutes on the topic at Blackhat, so come get your VirtSec boogie on there for a full explanation…

Alessandro got dinged in a comment on his blog for a statement in which he suggested that partially-collapsed as well as fully-collapsed DMZ’s with virtual separation of trust zones "…should be avoided at all costs because they imply the inviolability of the hypervisor (at any level: from the virtual networking to the kernel) something that nor VMware neither any other virtualization vendor can grant."

This appears contradictory to his initial assessment of DMZ virtualization wherein he stated that "…there [is] nothing bad in virtualizing the DMZ as long as we are fully aware of the risks."  In a way, I think I understand exactly where Alessandro is coming from, even if I don’t completely agree with him (or at least I partially do…)

This really paints an altogether unfortunate and yet accurate picture of the circular arguments folks engage in when they combine the following topics in a single argument:

• Securing virtualization
• Virtualizing security
• Security via virtualization

In the same way that we trust our operating system vendors who provide us with the operational underpinnings of our datacenters with the hope that they will approach a reasonable level of "security" in their products, we are basically at the same point with our virtualization (OS) platform providers.

Hope is not a strategy, but it seems we’ve at least accepted it for the time being… ;(

Sure there are new attack vectors and operational risks, but the slippery slope of not being able to really quantify whether you are more or less at risk based solely on the one-dimensional data point of the infallibility of the hypervisor  and then write the whole concept off seems a little odd to me.

If you’re truly assessing risk in the potential virtualization of your DMZ, you’ll take the operational/architectural guidelines as well as the subjective business impacts into consideration.  Simply stating that one should or should not virtualize a DMZ without a holistic approach is myopic.

To circle back on the topic, the choice of whether to — and how to — virtualize your DMZ  is really starting to gain traction.  I think the whitepaper took a decent first-pass stab at exploring how one might approach it, but the devil’s in the details — or at least the devil’s 4 horsemen are 😉

/Hoff

If you want to read about Montego Networks right off the bat, you can skip the Hoff-Tax and scroll down to the horizontal rule and start reading.  Though I’ll be horribly offended, I’ll understand…

I like being contradictory, even when it appears that I’m contradicting myself.  I like to think of it as giving a balanced perspective on my schizophrenic self…

You will likely recall that my latest post suggested that the real challenge for virtualization at this stage in the game is organizational and operational and not technical. 

Well, within the context of this post, that’s obviously half right, but it’s an incredibly overlooked fact that is causing distress in most organizations, and it’s something that technology — as a symptom of the human condition — cannot remedy.

But back to the Tech.

The reality is that for reasons I’ve spoken of many times, our favorite ISV’s have been a little handicapped by what the virtualization platforms offer up in terms of proper integration against which we can gain purchase from a security perspective.  They have to sell what they’ve got while trying to remain relevant all the while watching the ground drop out beneath them.

These vendors have a choice: employ some fancy marketing messaging to make it appear as though the same products you run on a $50,000+ dedicated security appliance will actually perform just as well in a virtual form.

Further, tell you that you’ll enjoy just as much visibility without disclosing limitations when interfaced to a virtual switch that makes it next to impossible to replicate most complex non-virtualized topologies. 

Or, just wait it out and see what happens hoping to sell more appliances in the meantime.

Some employ all three strategies (with a fourth being a little bit of hope.)

Some of that hoping is over and is on it’s way to being remedied with enablers like VMware’s VMsafe initiative.  It’s a shame that we’ll probably end up with a battle of API’s with ISV’s having to choose which virtualization platform providers’ API to support rather than a standard across multiple platforms.

Simon Crosby from Xen/Citrix made a similar comment in this article:

Officials at VMware-rival
Citrix Systems said they agreed that partnerships between security
vendors and virtualization tool providers are the way to go—as long as
everyone is on the same page. The objective is to work together with
security vendors and allow them to do their jobs across all
virtualization platforms, said Simon Crosby, chief technology officer
of the Virtualization and Management Division at Citrix.

“I would like VMware to
invite [Citrix] and Microsoft to that forum … so that there’s a common
set of APIs [that security vendors] can use to inspect against traffic,
inspect against memory and evoke security policy,” Crosby said. “I
think that’s an important thing to do. I don’t see any point in trying
to force vendors to retool for a different hypervisor.”

While I totally agree with his sentiment, I’m not sure Simon would be as vocal or egalitarian had Citrix been first out of the gate with their own VMsafe equivalent.  It’s always sad when one must plead for standardization when you’re not in control of the standards…and by the way, Simon, nobody held a gun to the heads of the 20 companies that rushed for the opportunity to be the first out of the gate with VMsafe as it’s made available.

While that band marches on, some additional measure of aid may come from innovative youngbloods looking to build and sell you the next better mousetrap.

As such, in advance of the RSA Conference in a couple of weeks, the security world’s all aflutter with the sounds of start-ups being born out of stealth as well as new-fangled innovation clawing its way out of up-starts seeking to establish a beachhead in the attack on your budget.

With the normal blitzkrieg of press releases that will undoubtedly make their way to your doorstop, I thought I’d comment on a couple of these companies in advance of the noise.

A lot of what I want to say is sadly under embargo, but I’ll get further in-depth later when I’m told I can take the wraps off.  You should know that almost all of these emerging solutions, as with the one below, operate as virtual appliances inside your hosts and require close and careful configuration of the virtual networking elements therein.

If you go back to the meat of the organization/operational issue I describe above, who do you think has access and control over the virtual switch configurations?  The network team?  The security team?  How about the virtual server admin. team…are you concerned yet?

Here’s my first Virtualized March Madness (VMM, get it!) ISV:

• 
  Montego Networks – John Peterson used to be the CTO at Reflex, so he knows a thing or two about switching, virtualization and security.  I very much like Montego’s approach to solving some of the networking issues associated with vSwitch integration and better yet, they’ve created a very interesting business model that actually is something like VMsafe in reverse. 

  Essentially Montego’s HyperSwitch works in conjunction with the integrated vSwitch in the VMM and uses some reasonably elegant networking functionality to classify traffic and either enforce dispositions natively using their own "firewall" technologies (L2-L4) or — and this is the best part — redirect traffic to other named security software partners to effect disposition. 

  If you look on Montego’s website, you’ll see that they show StillSecure and BlueLane as candidates as what they call HyperVSecurity partners.  They also do some really cool stuff with Netflow.

  Neat model.  When VMsafe is available, Montego should then allow these other third party ISV’s to take advantage of VMsafe (by virtue of the HyperSwitch) without the ISV’s having to actually modify their code to do so – Montego will build that to suit.  There’s a bunch of other stuff that I will write about once the embargo is lifted.

  I’m not sure how much runway and strategic differentiation Montego will have from a purely technical perspective as VMsafe ought to level the playing field for some of the networking functionality with competitors, but the policy partnering is a cool idea. 

  We’ll have to see what the performance implications are given the virtual appliance model Montego (and everyone else) has employed.  There’s lots of software in them thar hills doing the flow/packet processing and enacting dispositions…and remember, that’s all virtualized too.

  In the long term, I expect we’ll see some of this functionality appear natively in other virtualization platforms.

  We’ll see how well that prediction works out over time as well as keep an eye out for that Cisco virtual switch we’ve all been waiting for…*

I’ll be shortly talking about Altor Networks and Blue Lane’s latest goodies.

If you’ve got a mousetrap you’d like to see in lights here, feel free to ping me, tell me why I should care, and we’ll explore your offering.  I guarantee that if it passes the sniff test here it will likely mean someone else will want a whiff.

/Hoff

* Update: Alan over at the Virtual Data Center Blog did a nice write-up on his impressions and asks why this functionality isn’t in the vSwitch natively.  I’d pile onto that query, too.  Also, I sort of burned myself by speaking to Montego because the details of how they do what they do is under embargo based on my conversation for a little while longer, so I can’t respond to Alan…

Nurse, 10 cc’s of Adrenalin, stat!

As I mentioned in a prior posting, VMware’s VMsafe has the potential to inject life back into the atrophied and withering heart muscle of the security industry and raise the prognosis from DOA to the potential for a vital economic revenue stream once more.

How?  Well, the answer to this question really comes down to whether you believe that keeping a body on assisted life support means that the patient is living or simply alive, and the same perspective goes for the security industry.

With the inevitable consolidation of solutions and offerings in the security industry over the last few years, we have seen the commoditization of many markets as well as the natural emergence of others in response to the ebb and flow of economic, technological, cultural and political forces.

One of the most impacting disruptive and innovative forces that is causing arrhythmia in the pulse of both consumers and providers and driving the emergence of new market opportunities is virtualization. 

For the last two years, I’ve been waving my hands about the fact that virtualization changes everything across the information lifecycle.  From cradle to grave, the evolution of virtualization will profoundly change what, where, why and how we do what we do.

I’m not claiming that I’m the only one, but it was sure lonely from a general security practitioner’s perspective up until about six months ago.  In the last four months, I’ve given two keynotes and three decently visible talks on VirtSec, and I have 3-4 more tee’d up over the next 3 months, so somebody’s interested…better late than never, I suppose.

How’s the patient?

For the purpose of this post, I’m going to focus on the security implications of virtualization and simply summarize by suggesting that virtualization up until now has quietly marked a tipping point where we see the disruption stretch security architectures and technologies to their breaking point and in many cases make much of our invested security portfolio redundant and irrelevant.

I’ve discussed why and how this is the case in numerous posts and presentations, but it’s clear (now) to most that the security industry has been clearly out of phase with what has plainly been a well-signaled (r)evolution in computing.

Is anyone really surprised that we are caught flat-footed again?  Sorry to rant, but…

This is such a sorry indicator of why things are so terribly broken with "IT/Information Security" as it stands today; we continue to try and solve short term problems with even shorter term "solutions" that do nothing more than perpetuate the problem — and we do so in a horrific display of myopic dissonance, it’s a wonder we function at all.   Actually, it’s a perfectly wonderful explanation as to why criminals are always 5 steps ahead — they plan strategically while acting tactically against their objectives and aren’t afraid to respond to the customers proactively.

So, we’ve got this fantastic technological, economic, and cultural transformation occurring over the last FIVE YEARS (at least,) and the best we’ve seen as a response from most traditional security vendors is that they have simply marketed their solutions slimly as "virtualization ready" or "virtualization aware" when in fact, these are simply hollow words for how to make their existing "square" products fit into the "round" holes of a problem space that virtualization exposes and creates.

Firewalls, IDS/IPSs, UTM, NAC, DLP — all of them have limited visibility in this rapidly "re-perimeterized" universe in which our technology operates, and in most cases we’re busy looking at uninteresting and practically non-actionable things anyway.  As one of my favorite mentors used to say, "we’re data rich, but information poor."

The vendors in these example markets — with or without admission — are all really worried about what virtualization will do to their already shrinking relevance.  So we wait.

Doctor, it hurts when I do this…

VMSafe represents a huge opportunity for these vendors to claw their way back to life, making their solutions relevant once more, and perhaps even more so.

Most of the companies who have so far signed on to VMsafe will, as I mentioned previously, need to align roadmaps and release new or modified versions of their product lines to work with the new API’s and management planes. 

This is obviously a big deal, but one that is unavoidable for these companies — most of which are clumbsy and generally not agile or responsive to third parties.  However, you don’t get 20 of some of the biggest "monoliths" of the security world scrambling to sign up for a program like VMsafe just for giggles — and the reality is that the platform version of VMware’s virtualization products that will support this technology aren’t even available yet.

I am willing to wager that you will, in extremely short time given VMware’s willingness to sign on new partners, see many more vendors flock to the program.  I further maintain that despite their vehement denial, NAC vendors (with pressure already from the oncoming tidal wave of Microsoft’s NAP) will also adapt their wares to take advantage of this technology for reasons I’ve outlined here.

They literally cannot afford not to.

I am extremely interested in what other virtualization vendors’ responses will be — especially Citrix.  It’s pretty clear what Microsoft has in mind.  It’s going to further open up opportunities for networking vendors such as Cisco, f5, etc., and we’re going to see the operational, technical, administrative, "security" and governance lines  blur even further.

Welcome back from the dead, security vendors, you’ve got a second chance in life.  I’m not sure it’s warranted, but it’s "natural" even though we’re going to end up with a very interesting Frankenstein of a "solution" over the long term.

The Doctor prescribes an active lifestyle, healthy marketing calisthenics, a diet with plenty of roughage, and jumping back on the hamster wheel of pain for exercise.

/Hoff

Back in August before VMworld 2007, I wrote about the notion that given Cisco’s investment in VMware, we’d soon see the ability for third parties to substitute their own virtual switches for VMware’s. 

Further, I discussed the news that VMware began to circulate regarding their release of an API originally called Vsafe that promised to allow third party security and networking applications to interact with functions exposed by the Hypervisor. 

Both of these points really put some interesting wrinkles — both positive and potentially negative — in how virtualization (and not just VMware’s) will continue to evolve as the security and networking functions evolve along with it.

What a difference a consonant makes…

Several months later, they’ve added the consonant ‘m’ to the branding and VMsafe is born.  Sort of.  However, it’s become more than ‘just’ an API.  Let’s take a look at what I mean.

What exactly is VMsafe? Well, it’s a marketing and partnership platform, a business development lever, an architecture, a technology that provides an API and state of mind:

VMsafe is a new program that leverages the properties of VMware Infrastructure to protect machines in ways previously not possible with physical machines. VMsafe provides a new security architecture for virtualized environments and an application program interface (API)-sharing program to enable partners to develop security products for virtualized environments.

What it also provides is a way to make many existing older and consolidating security technologies relevant in the virtualized world given that today their value is suspect in the long term without it:

The VMsafe Security Architecture provides an open approach that gives security vendors the ability to leverage the inherent properties of virtualization in their security offerings.

Customers running their businesses on VMware Infrastructure will be assured that they are getting the best protection available – even better than what they have on physical infrastructure.

VMsafe adds a new layer of defense that complements existing physical security solutions such as network and host protection, shielding virtual machines from a variety of network, host and applications threats. This additional layer of protection can help enterprise organizations to dramatically increase the security posture of their IT environments.

There is then hope that a good deal of the visibility we lost in the limited exposure existing security solutions have across virtualized environments, we may get back.

Of course, this good news will be limited to those running VMware’s solutions and re-tooled versions of your favorite security vendor’s software.  What that means for other virtualization platforms is a little more suspect.  Since it requires the third party software to be re-written/adapted to use the VMsafe API, I can’t see many jumping on the wagon to support 3-4 VMM platforms.

Ack!  This is why we need a virtualization standard like OVF and a cross-platform signaling, telemetry and control plane definition!

So how does VMsafe work?

VMsafe enables third-party security products to gain the same visibility as the hypervisor into the operation of a virtual machine to identify and eliminate malware, such as viruses, trojans and key-loggers. For instance, security vendors can leverage VMsafe to detect and eliminate malware that is undetectable on physical machines. This advanced protection is achieved through fine-grained visibility into the virtual hardware resources of memory, CPU, disk and I/O systems of the virtual machine that can be used to monitor every aspect of the execution of the system and stop malware before it can execute on a machine to steal data.

…

VSAFE enables partners to build a virtualization-aware security solution in the form of a security virtual machine that can access, correlate and modify information based on the following virtual hardware:

• Memory and CPU: VMsafe provides introspection of guest VM memory pages and cpu states.
• Networking: Network packet-filtering for both in-hypervisor and within a Security VM.
• Process execution (guest handling): in-guest, in-process APIs that enable complete monitoring and control of process execution.
• Storage: Virtual machine disk files (VMDK) can be mounted, manipulated and modified as they persist on storage devices.

So where do we go from here?

With nothing more than a press release and some flashy demos, it’s a little early to opine on the extensibility of VMsafe, but I am encouraged by the fact that we will have some more tools in the arsenal, even if they are, in essence, re-branded versions of many that we already have.

However, engineering better isolation combined with brokered visibility and specific authorized/controlled access to the VMM is both a worthy endeavor that yields all sorts of opportunities, but given my original ramblings, makes me a bit nervous.

Alessandro from the virtualization.info blog gave us a description of the demo given at VMworld Europe that illustrates some of this new isolation and anti-malware capability:

A Windows XP virtual machine gets attacked with a malicious code that
copies away corporate documents but another virtual machine with
security engine is able to transparently recognize (a virtual memory
scan through VMsafe APis access) the threat and stop it before it
compromises the guest OS.

Steps in the right direction, for sure.  Since details regarding the full extent of anti-malware capabilities are still forthcoming, I will reserve judgment until I get to speak with Nand @ VMware to fully understand the scope of the capabilities.

I am sure we will see more claims surface soon suggesting with technology such as this will produce virtualized environments that are "more secure" than their non-virtualized counterparts.  The proof is in the pudding, as they say.  At this point, what we have is a very tantalizing recipe.

I’d suggest we’ll also see a fresh batch of rootkit discussions popping up soon…I’d really like to see the documentation surrounding the API.  I wonder how much it costs to sign up and be authorized to participate with VMsafe?

Some of the Determina functionality is for sure making its way into VMsafe and it will be really interesting to see who will be first first out of the gate to commercially offer solutions that will utilize VMsafe  once it’s available — and it’s a little unclear when that will be.

Given who demoed at VMworld Europe, I’d say it’s safe to bet that McAfee will be one of the first along with some of the more agile startups that might find it easier to include in their products.  The initial lineup of vendor support makes up some of the who’s-who in the security space — all except for, um, Cisco.  Also, where the heck is SourceFire?

When do the NAC and DLP vendors start knocking?

More detailed analysis soon.

/Hoff

The virtualization security (VirtSec) FUD meter is in overdrive this week…

Part I:
     So, I was at a conference a couple of weeks ago.  I sat in on a lot of talks.
     Some of them had demos.  What amazed me about these demos is that in
     many cases, in order for the attacks to work, it was disclosed that the
     attack target was configured by a monkey with all defaults enabled and no
     security controls in place.  "…of course, if you checked this one box, the
     exploit doesn’t work…" *gulp*

Part II:
     We’ve observed a lot of interesting PoC attack demonstrations such as those at shows being picked
     up by the press and covered in blogs and such.  Many of these stories simply ham it up for the
     sensational title.  Some of the artistic license and innacuracies are just plain recockulous. 
     That’s right.  There’s ridiculous, then there’s recockulous. 

Example:
     Here’s a by-line from an article which details the PoC attack/code that Jon Oberheide used to show
     how, if you don’t follow VMware’s (and the CIS benchmark) recommendations for securing your
     VMotion network, you might be susceptible to interception of traffic and bad things since — as
     VMware clearly states — VMotion traffic (and machine state) is sent in the clear.

     This was demonstrated at BlackHat DC and here’s how the article portrayed it:

Jon Oberheide, a researcher and PhD candidate at the
University of Michigan, is releasing a proof-of-concept tool called
Xensploit that lets an attacker take over the VM’s hypervisor and
applications, and grab sensitive data from the live VMs.

     Really?  Take over the hypervisor, eh?  Hmmmm.  That sounds super-serious!  Oh, the humanity!

However, here’s how the VMTN blog rationally describes the situation in a measured response that does it better than I could:

Recently a researcher published a proof-of-concept called
Xensploit which allows an attacker to view or manipulate a VM undergoing live
migration (i.e. VMware’s VMotion) from one server to
another. This was shown to work with
both VMware’s and Xen’s version of live migration. Although impressive, this work by no means
represents any new security risk in the datacenter. It should be emphasized this proof-of-concept
does NOT “take over the hypervisor” nor present
unencrypted traffic as a vulnerability needing patching, as some news
reports incorrectly assert. Rather, it a
reminder of how an already-compromised network, if left unchecked, could be
used to stage additional severe attacks in any environment, virtual or
physical. …

Encryption of all data-in-transit is certainly one well-understood mitigation
for man-in-the-middle attacks.  But the fact
that plenty of data flows unencrypted within the enterprise – indeed perhaps
the majority of data – suggests that there are other adequate mitigations. Unencrypted VMotion traffic is not a flaw,
but allowing VMotion to occur on a compromised network can be. So this is a good time to re-emphasize hardening best practices for VMware
Infrastructure and what benefit they serve in this scenario.

I’m going to give you one guess as to why this traffic is unencrypted…see if you can guess right in the comments.

Now, I will concede that this sort of thing represents a new risk in the datacenter if you happen to not pay attention to what you’re doing, but I think Jon’s PoC is a great example of substantiating why you should follow both common sense, security hardening recommendations and NOT BELIEVE EVERYTHING YOU READ.

If you don’t stand for something, you’ll fall for anything.

/Hoff