Rational Survivability

I’ve spent quite a bit of time investigating emerging technology solutions for virtualization security (VirtSec) lately.  I’ve made mention of an idea that conceptually didn’t gel until this last week.

I was speaking at TechTarget’s Financial Information Security Decisions show in New York and was paired up in the network workshop with Joel Snyder of Opus One.

Joel was presenting his 5 myths of Information Security and one of the myths was (paraphrasing) that Intrusion Detection solutions don’t detect solutions. 

What Joel went on to suggest is that what IDS solutions actually do is provide one with a perspective visibility across the network; determining what represents an actual "intrusion" is a contextual argument that goes to the efficacy and correlation capabilities of the platform(s.)

This got me thinking along the lines of some of the emerging IDP (intrusion detection and prevention) solutions from emerging vendors in the virtualization space.

Something rather profound but obvious dawned on me.

Given the integration for management of these "security" solutions with the management platforms of the virtualization platform providers AND the operational shift of who was managing the security solutions (see here) really means that these aren’t really virtualization security solutions at all, they are actually vitualization visualization solutions.

Virtualization management platforms provide the configuration and operational telemetry regarding the virtual environment to these solutions which does what most HostSec or NetSec solutions have been unable to do in the past: gain context regarding how the infrastructure the security solutions are protecting are actually configured.

HostSec and NetSec solutions have no context of the solutions outside of the host they are protecting or the network segment/IP address they are connected to respectively.  Not so with VirtSec solutions.

That’s pretty neat when you think of it.  Even though we’re substantially handicapped as to what these solutions can *do* with this capability today (see here) integrating this capability can dramatically and positively affect the way in which "security" administration and analytics manifests themselves over time.

"Yeah, but these are basically the same views someone might get looking at a firewall, IDS or IPS tool today," you might argue.  That’s right, except we already know that server and virtualization administrators (as well as most network folk) don’t have access to those tools…

So in many cases the administrators who will be looking at this information are not "security" folks by trade, so the (and you’ll excuse the wording) dumbing down of this information actually provides a very good perch upon which to troubleshoot and extend the forced simplicity of "checkbox" security in the virtualization platforms to this new class of security administrator.

This may be the first time some of these teams have had access to "security" telemetry of this kind.

In the long term, he challenge will be how, when you have multiple of these solutions, you gain a consolidated view, but the reality is that the NetSec and HostSec admins can use this same view and then click-through into the specific toolset management stacks for finer-grained configuration/analysis. 

This is actually an interesting way to think about how the re-integration of the server admins, network and security teams might become more cohesive operationally in the future…through the same lens of visualizing the environment.

Here are some ideas of what I’m talking about; these are some snapshots of management interfaces of upcoming VirtSec solution providers.  These are random shots of some of the different views of managing virtual appliances…

Altor:

Blue Lane:

Catbird:

Reflex:

Thanks to Amir-Ben Afraim (Altor,) Greg Ness (Blue Lane,) Michael Berman (Catbird,) and Dave Devalk (Reflex) for getting these images to me.  Also, hat-tip to Joel Snyder for the noodle nudge…

/Hoff

Tim Greene from Computerworld wrote a story last week titled "Security software makes virtual servers a moving target." 

This story draws attention to a story on the same topic that popped up a while ago (see Dark Reading) about some research led by George Mason University professor Arun Sood that is being productized and marketed as "Self Cleansing Intrusion Tolerance (SCIT)"

SCIT is based upon the premise that taking machines (within a virtualized environment) in and out of service rapidly and additionally substituting the underlying operating systems/application combinations reduces the exposure of attack and hastens the remediation/mitigation process by introducing the notion of what Sood calls "security by diversity."

Examples are given in the article suggesting the applicability of application types for SCIT:

SCIT is best suited to servers with short transaction times and has been tested with DNS, Web and single-sign-on servers, he says, which can perform effectively even if each virtual server is in use for just seconds.

In today’s multi-tier, SOA, web2.0, cloud-compute, mashup world, with or without the issue of preservation of state across even short-transactional applications, I’m not sure I see the practical utility in this approach.  The high-level concept, yes, the underlying operational reality…not so much.

Some of you might notice the, um, slightly different comparative version of Sood’s acronym reflecting my opinion of this approach in this blog entry’s title… 😉

I think that SCIT’s underlying principles lend themselves well to the notions I champion of resilient and survivable systems, but I think that the mechanical practicality of the proposed solutions — even within the highly dynamic and agile framework of virtualization — simply aren’t realistic today.

Real-time infrastructure with it’s dynamic orchestration, provisioning, governance, and security is certainly evolving and we might get to the point where heterogeneous systems are autonomously secured based upon global policy definitions up and down the stack, but we are quite some time away from being able to realize this vision.

You will no doubt notice that the focal element of SCIT is the concept of a security-centric perspective on lifecycle management of VM’s.  It’s quite obvious that VM lifecycle management is a hotly-contested topic for which many of the large infrastructure players are battling. 

Security will simply be a piece of this puzzle, not the focus of it.

This is not to say that this solution is not worthy of consideration as we look out across the horizon, and from a timing perspective it will likely surface again given it’s "ahead of it’s deployable time" status but I’m forced to consider what box I’d check in describing SCIT today:

• Feature
• Solution
• Future

Neat stuff, but if you’re going to take investment and productize something, it’s got to be realistically deployable.  I’d suggest that baking this sort of functionality into the virtualization platforms themselves and allowing for universal telemetry (sort of like this) to allow for either "self cleansing intrusion tolerance" or even "self healing intrusion tolerance" is probably a more reasonable concept. 

/Hoff

Mark Gaydos from Tripwire’s blog wrote an interesting article titled "Ops or Security: Who’s Responsible for Securing Virtualization?"  The outcome is pretty much inline with my prior points that the biggest challenges we have in virtualization are operational and organizational rather than technical.

To wit, I quoteth from Mark’s post:

Tripwire recently performed a 25 question survey on virtualization security.  Respondents broke down 78%/22% between management and administrator/staff respectively.  We will be publishing a report around this survey in the next two weeks. 

However, one of the interesting points that came out of the survey was that respondents feel that the operations team is responsible for securing a virtualized environment (almost two thirds of the respondents felt this way).  This includes over half of the actual  “security” personnel who took the survey who feel operations has this responsibility. 

That’s right!  Over half of the people covering security who responded to the survey said operations needs to secure virtual systems and not them.

My question is why?  Does security not want to deal with virtualization?  Do personnel feel that operations is closer to virtualization and they understand the issues?  Does security just want to wash their hands of the issue?  Or is management just leaning towards having operations handle everything around virtualization?

However, I wonder how much Mark read into the security personnel’s answers inasmuch as he suggests that they do "…not want to deal with virtualization" versus perhaps the fact that they don’t actually have the visibility or access to the tools to do so!*

Responsibility versus desire are two very different things!

Managing the "security" of virtualized environments today really centers around the deployments of virtual appliances and the configuration of the vSwitches.  That means in a VMware environment, you have to have access and rights via Virtualcenter.  The same is true in terms of Xen derivatives; if you don’t have access to configure and provision the networking and VM’s, you’re done.

Security in virtualized environments today is literally often thought of as a checkbox or two in a GUI somewhere.  (All things considered, it would be great to be able to realize that one day…)

Just like security folks have locked server and network admins out of *their* firewalls and IPS’s, and as network folks have done the same in *their* routers and switches, virtual SysAdmins have done the same in *their* virtual server environments.  If you don’t have access to the VM command and control, you can’t manage the security bits and pieces bolted onto it.

I don’t think it’s that the security folks *want* to surrender the responsibility, I think it’s that they never had it in the first place the moment the V-word entered the picture.

It ain’t rocket science.  It ain’t voodoo.  It ain’t a tectonic buck-passing conspiracy.  It’s access, separation of duties (by force,) visibility and capability, plain and simple.

/Hoff

*Update: Per Amrit’s excellent comments, I look forward to Tripwire releasing the report to gain clarity on the question(s) asked as it begs the point as to whether the answers Mark refers to were in regards to the mechanical operationalization of security (the "doing" part) or the policy, strategy, audit and monitoring  tasks.  Are we talking about "security management" in general or "security operations?"

In either circumstance the "security" team is — based upon my observation from feedback — being left out of both.

From my good friends over at Tripwire…

I haven’t been able to try ConfigCheck out myself yet, but reports from a couple of trusted sources have suggested it’s a fantastically useful tool, and you can’t beat the price as it’s FREE!

Tripwire® ConfigCheckTM is a free utility that rapidly assesses the security of VMware ESX 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines. Developed by Tripwire in cooperation with VMware, Tripwire ConfigCheck ensures ESX environments are properly configured—offering immediate insight into unintentional vulnerabilities in virtual environments—and provides the necessary steps towards full remediation when they are not.

If I have time next week, I plan to give this a whirl, but I’d suggest that if you’ve already implemented VMware or are planning to, you should make use of a utility such as this…until it’s bundled into the platforms themselves 😉

Get your copy here.

Good move by Tripwire.

If Barracuda attempting to gobble up SourceFire today wasn’t interesting enough, check this out…

WALTHAM, Mass., May 30 /PRNewswire/ — Hyperbole, Inc., the the pioneer and leader in virtualization security solutions today announced it has emerged from stealth mode and raised $14 million in a Series A funding which it will use to expand its R&D efforts and grow its sales and distribution teams.

Hyperbole’s flagship product, HyperTension, provides a zero footprint and forensically tight paradigm-shift in the emerging virtualization security (VirtSec) market by automatically protecting all virtual infrastructure against known or unknown attacks without the need for expensive and clumsy IDS, firewall and IPS technology. 

With no agent software and no hardware requirements save for a specially-constructed tamper-proof USB device called the HyperDrive, HyperTension is able to secure any virtualization platform automatically within seconds and with no downtime required.

HyperTension provides an undetectable ring compression insertion technology that injects itself into memory space transparently and utilizes the flash memory space available in PCI cards present in the system to load, thereby not corrupting the main heap and rendering itself undetectable. 

Further, HyperTension will probe for the presence of parallelized graphics processing units (GPU) from leading graphics card providers and if found, will utilize them to provide the compute cycles necessary for operation thereby not impacting the on-board main CPU or cache, further lessening the impact of the solution running in virtualized environments. 

This allows for massive computation capabilities used to provide real-time memory-space attack detection functionality which can be manually or automatically adjusted using our patented HyperSensitivity comb filter technology.

Hyperbole’s patented HyperVentilation technology utilizes quantum cryptography and open source algorithms to create "holes" in memory to dynamically encrypt/decrypt the entire memory space of a virtualized host and upon register access, leverage commodity TPM solutions to authenticate and decrypt memory on the fly when used in conjunction with any of Hyperbole’s partner-supplied whitelisting solutions.

Once accessed, HyperTension automatically performs an ASLR operation for pointer obfuscation and then re-encrypts the memory space using a newly-generated quantum key derived from the unique properties of the hashed cache entries from the rotating cipher.

This provides unbreakable security since only authorized applications can attempt to gain access to HyperVentilated memory space which is also encrypted to prevent unauthorized access.

…

Speechless. 

/Hoff

My last couple of VirtSec posts have caused quite a stir in certain circles.

The “debate” between who “owns” VirtSec that originated as part of my response to Simon Crosby of Citrix regarding the same has been picked up and amplified on multiple fronts.

Greg Ness from BlueLane wrote a piece referencing it that was cross-posted on virtualization.com and that even made its way up to VC/investment blogs such as seekingalpha.com (Citrix vs. Chris Hoff 😉 and has had my mobile ringing/vibrating itself off my desk over the last week or so.

It’s hard to believe sometimes just how many people — and who — reads my steaming pile of blogginess.

The second post of interest was in regard to the provenance of VMware’s VMsafe and my reflection on prior art (Livewire) by VMware’s Rosenblum & Garfinkel which seems as though it could be the progenitor of the upcoming technology.

The very tail-end update of that post referenced another piece of research produced by Komoku based upon similar work focused on rootkit defense. As I pointed out, Komoku was recently acquired by Microsoft.

I added those comments deliberately as a parenthetical — almost like a bookmark — because what I intended to do next was directly compare and contrast the technology architectures and approaches of VMware, Citrix and Microsoft as it relates to security integration.

It seems a bunch of really bright folks caught onto that because a slew of links (such as this one) followed — driven mostly by Alessandro’s (virtualization.info) post titled “Is Microsoft Working On VMsafe-like Framework”

I think that’s an excellent question 😉

It’s pretty clear where Citrix’s CTO stands on the matter — as flawed as I see his shortsighted market approach (note I didn’t say *technical approach*) — but Microsoft stands to gain an interesting foothold in regards to security should they play this game correctly.

I found it interesting that others are starting to recognize that the virtualization battle isn’t going to be won by a shoot-out and the hypervisor-version of the OK corral. It’s the effectiveness of the ecosystem and the ability for the channel to serve it up and the customers to implement it.

People are sick of sweeping up the decaying corpses of good technical solutions that suck in terms of integration, implementation, operationalization and accountable support — especially when they have to keep paying for it. Ah the “best-in-breed” versus “good-enough” debate again?

Not to further pick on Citrix (or Xen specifically) but here’s a great post from Schley Andrew Kutz from the searchservervirtualization.com blog titled “Xen: An endangered species in the virtualization ecosystem?“:

While Citrix Systems’ Xen’s ubiquity may help the technology earn a legacy as the invisible hypervisor, it may also prove the most challenging next step for IT administrators and developers who want to find or develop software that leverages, supports or extends the Xen hypervisor.

…

While ultimately it may not prove difficult to develop cutting-edge technology compatible with the Xen hypervisor, it may prove so to market it. If you are in the business of selling virtualization add-on products, you want to ensure that your product is compatible with VMware Infrastructure, because that is where the sales are.

…

As Xen’s legacy may be to become the ubiquitous, embedded hypervisor for all to use, its strength may also be its greatest detriment to Xen-based virtualization platforms. Xen’s strength is its practical application as the invisible, reused, resold, embedded hypervisor, but invisibility just hasn’t worked in Citrix’s favor. Instead, it shields partners from building ecosystems around Xen and has marginalized the brand name.

Amen to that.

Take heed, Citrix. I maintain your CTO is blinded by what can only be described as a denial of market realities and an undying (arrogant) allegiance to what some might consider to be an architecturally superior product on some fronts, but a lacking solution on many others.

Securing the hypervisor is definitely important. However, securing both the hypervisor and the assets that sit on top of it by providing the most extensible, effective and manageable means of doing so is really what’s important to customers. Sometimes, it has to be about more than where you came from. Sometimes it’s about where you’re going.

I’ll be finishing up my post on where I think Microsoft ought to go shortly.

/Hoff

One of the things I try to do when looking forward for inspiration in solving problems is to ensure that I spend enough time looking back to gain perspective.  I’ve been thinking a lot about models for virtualization security lately.

As I surveyed the options (or lack thereof) splayed about before me in terms of deployment options and available technology to solve some of the problems I’ve been researching, I was struck by what I can only describe as a ghost of future’s past. 

It shouldn’t really surprise me like it does, but I always giggle when reminded of my own favorite saying: "Security is like bellbottoms — every 20 years or so, the same funny-looking kit comes back into style."

As it is with jeans, it is with security solutions.

I dredged up some of my collected research from moon’s ago on the topic and dusted off a PDF that I had completely forgotten about as I was trying to piece together some vague semblance of something that strangely reminded me of VMware’s VMsafe.

I cracked a gigantic smile when I saw the authors — Tal Garfinkel and some guy named Mendel Rosenblum (now co-founder and chief scientist at VMware.)

The PDF in question is titled Virtual Machine Introspection ("productized" as LiveWire) and presents the following case:

In this paper we present a new architecture for building intrusion
detection systems that provides good visibility into the state of the
monitored host, while still providing strong isolation for the IDS,
thus lending significant resistance to both evasion and attack.  

Our approach leverages virtual machine monitor (VMM) technology. This mechanism allows us to pull our IDS “outside” of the host it is monitoring, into a completely
different hardware protection domain, providing a high-confidence
barrier between the IDS and an attacker’s malicious code.

We achieve this through the use of a virtual machine monitor. Using this approach allows us to isolate the IDS from the monitored host but still retain excellent visibility into the host’s state. The VMM also offers us the unique ability to completely mediate interactions between the host software and the underlying hardware. We present a detailed study of our architecture, including Livewire, a prototype implementation. We demonstrate Livewire by implementing a suite of simple intrusion detection policies and using them to detect real attacks.

I got to thinking about the relevance of this approach because of some of the arguments that Simon Crosby made in our debate recently.  I wanted to spend some more time thinking about the architectural differences between VMware and Xen so I could try an appreciate the genesis of Simon’s comments in context.

This paper and the Livewire prototype was created circa 2002.  It’s six years later and we’re just now starting to see products and technology being announced as "new and fresh"  that is basically just like Livewire.

While it’s certainly not the first and only research on this topic, it’s interesting to see that sometimes the wisdom of the past just takes just a little longer to cook before it’s fully baked, ready for icing and ready to be consumed.

If VMsafe is an example of the evolution of prior art like Livewire, what else do we have to look forward to that’s buried somewhere waiting to come back to life?  Oh wait, those mainframes are coming back, aren’t they?  What’s old is new again.

/Hoff

{Update: I also found some cool related stuff from Tim Fraser called Virtual Machine Introspection for Cognitive Immunity (kernel rootkit mitigation using VM Introspection) from Komoku which was acquired about a month ago by, gasp, Microsoft…}

It seems I have fallen victim to a series of misunderstandings these days.

First there was Joanna-Gate and now Simon Crosby, Citrix’s CTO, suggests in a blog entry titled "Chris Hoff & The Mother Of All Misunderstandings" that I’m puffing on the wrong end of my cigars for disagreeing with his position.

I’m a little concerned that Simon’s response to me was issued on what is listed as the "beta" version of Citrix’s official blog.  Perhaps the virtualized version hasn’t made it out of QA yet? 😉

Simon’s response was extremely well crafted to avoid responding to most of my actual points, was contextually oblique at points, and was a fantastic marketing piece for Xen Citrix, but I wish he’d paid more attention to the actual points within my post. 

Further his little quips/comments on his hyperlinks "Who is this guy, anyway?  Think before you type dude, we’re not idiots," etc. didn’t go unnoticed – cute but juvenile)

I am, however, honored that Simon would accord me the high-status of being "…normally fairly clued-in:"

I
reckon that Hoff, who is normally fairly clued-in,  has put the smoking
end of the cigar in his mouth before thinking through this argument.
He’s horribly confused, but as smug as always, so let me clarify what I
said, and what it means.

…but I can assure you that I’ve only ever done that with a cigar once,
and it was for a much better reason than blogging.  If you must know,
it was Kentucky’s finest bourbon.  That is all I’m going to say about
that. 

I’m glad he’s "clarifying" what he said, since I will also.  I seem to have that effect on people.  Must be the accent thing…

The reason for my allergic reaction to Simon’s comments stem from my opinion that it is the responsibility of virtualization platform providers to ensure that their "[virtualized] data center operating system platforms of the future" don’t become the next generation of insecure infrastructure.

Simon sums up his opinion:

In summary an assertion that the virtualization platform vendor has
to fix the sad state of the OS/App world by making it secure is
demanding too much.  It would mean that we have to be experts in every
piece of system software including all of the vulnerabilities of all
OSes and their apps.  In my view the reason the state of security is
poor now is because of the monolithic approaches of traditional OS and
app vendors. 

We will focus manically on our layer, make it
secure, tiny and bulletproof to attack in its own right.  And we will
work closely with experts in security of OSes and Apps to give them an
opportunity to implement guest-level security outside the guest,
through privileged interfaces that themselves are secure.

After 15 years of dealing with this crap, I respectfully suggest that it is not too much to ask and it’s about time we stood up and did.  First  you criticize OS/App. vendors and blame them for the state of security because of their "monolithic approach" and then you go on to propose the exact same thing!

Focusing only on your little patch of grass is short-sighted and it won’t work.  Just like it hasn’t worked in the past.  It’s a disaster waiting to happen, and you’re enabling it. 

I shudder at the potential tunnel vision of virtualization platform providers only focusing on the security of the hypervisor without taking the bigger picture into consideration and expect a piecemeal approach to securing the expanse of the virtualized environment to suffice.

It’s clear you’re making arguments about security from an engineering and code-base perspective that is simply disconnected from the realities of what it means to actually deploy these solutions. 

Virtualization is more than just the hypervisor.  You should know that by now, Simon.  The company that acquired your company knows all about that.  The hypervisor will shortly become a commodity, so in the long term the value brought to bear has to be more than just an ultra-thin layer of code:

…and furthermore, we’re going to deploy many of them:

I wish to make it clear that I hold all virtualization platform vendors to the same level of scrutiny and criticism, not just Citrix. 

I happen to like Xen very much.  I like VMware, also.  I think the latter is more realistic and measured when it comes to addressing the need and approach in recognizing that as a major layer in the infrastructure, there’s more required than to just secure the hypervisor and leave the remaining mess to someone else to solve.

I think Simon’s blog title is apropos, but I think the misunderstanding is his.

It’s important to understand that I’m not suggesting that virtualization platform providers should secure the actual guest operating systems
but they should enable an easier and more effective way of doing so when virtualized.

I mean that the virtualization platform providers should ensure the security of the instantiation of those
guests as "hosted" by the virtualization platform.  In some cases this means leveraging technology present in the virtualization platform to do things that non-virtualized instances cannot. That’s more than just securing the hypervisor.

Securing the hypervisor whilst closing your eyes to the likelihood
that the majority of attacks against it and other guests will come from "guests" within the same system is planting your head in the sand.  That means that there will be a need to ensure that certain behaviors specific to the hosted guests are mitigated to ensure that bad things don’t happen — to the guest or the hypervisor.

Transferring the responsibility to secure the environment to third party security ISV’s in order
to secure the VM’s
and preventing them from compromising one another or the hypervisor is
difficult for me to comprehend, especially when they are playing catch up of what virtualization means within the context of security.

Fundamentally, attempting to mate static and topology-dependent policies to incredibly dynamic and transitive technology delivered by virtualization will simply fail.  Third party security ISV’s will simply require a complete re-tool to even get close to delivering this and will need to provide intimate hooks to allow for this policy/guest affinity to occur in the first place.

I consider the virtualization infrastructure layer as that of an operating system and as such, I would expect that the underpinning mechanicals are as sound and secure as possible while also ensuring that anything running on top of it is as secure as possible, also.

Let’s take Microsoft (with or without Hyper-V) as an example:

Microsoft is fundamentally concerned now with making the OS as
resilient and secure as possible whilst preventing the applications and
interaction with elements riding on top of the OS from doing bad things
to the system as a whole; this isn’t just to protect the OS, but the
assets on it. 

This is really what I’m getting at.  Yes, Microsoft is an OS provider.  Shortly, that OS provider will integrate virtualization directly into the operating system.  That means more, not less, direct integration and security embedded as a function of the virtualization platform.  Citrix, VMware, etc. are all just operating system vendors of a different shape and size.

It’s unclear to me, Simon, whether your arguments are meant to justify a business model, a lack of planning, a crafty plan to perpetuate the security hamster wheel of pain, or all of the above.  It’s clear to me, however, that you’ve not felt the pain of actually having to use the products you suggest should be deployed in order to secure this mess.

I promised myself I wouldn’t turn this into one of those cut/paste blog pong entries, but the following really confused me:

But we are not in the business of specifically securing guests or their
applications, other than through offering a secure virtualization
platform.  Even VMware with VMsafe simply exposes APIs to third party
security vendors, so that customers can choose their preferred security
partner to secure guests.  I think that the VMware Determina
acquisition was very smart, and that hints to me that VMware sees
itself having a greater role in the security of guest OSes, since it
could choose to be in the vulnerability checking business without 3rd
party security vendors, but thus far they are working very openly with
the ecosystem.

So which is it?  You’ve established that Citrix is not in the business of securing guests or applications (you must mean Xen specifically, because somebody at Citrix spent quite a bit of money on this stuff with their other acquisitions) and that you believe it to be a lousy idea, but you think that VMware’s approach through their Determina acquisition as well as the capabilities of VMsafe is "…very smart?"

Simon, you’re the CTO and I’m the security wonk.  If we didn’t disagree, I’d be alarmed.  However, I think you might want to rethink your approach to how you market the security of your platform.

I’ve got a cigar for you anytime you want one.  I’ll let you light it.

/Hoff

In an article over at SearchSecurity.com, Simon Crosby, the CTO of Citrix, suggests that "Virtualization vendors [are] not in the security business." 

Besides summarizing what is plainly an obvious statement of fact regarding the general omission of integrated security (outside of securing the hypervisor) from most virtualization platforms, Crosby’s statement simply underscores the woeful state we’re in:

While virtualization vendors will do their role in protecting the hypervisor, they are not in the business of catching bad guys or discovering vulnerabilities, said Simon Crosby, chief technology officer of Citrix Systems.

Independent security vendors will play a critical role in protecting virtual environments, he said. "The industry has already decided a long time ago that third party vendors are required to secure any platform," Crosby said. In this interview, Crosby agrees that using virtual technology introduces new complexities and security issues.

He said the uncertainties will be addressed once the industry matures.

I’m sure it’s reasonable to suggest that nobody expects virtualization platform providers to "…catch
bad guys," but I do expect that they employ a significant amount of
resources and follow an SDLC to discover vulnerabilities — at least in
their software.

Further, I don’t expect that the hypervisor should be the place in which all security functionality is delivered, but simply transferring the lack of design and architecture forethought from the hypervisor provider to the consumer by expecting someone else to clean up the mess is just, well, typical.

I love the last line.  What a crock of shit.  We’ve seen how well
this approach had worked with operating system vendors in the past, so why
shouldn’t the "next generation" of OS vendors — virtualization
platform providers — follow suit and not provide for a secure operating environment?

Let’s see, Microsoft is investing hugely in security.  Cisco is too.  Why would the other tip of the trident want to?  VMware’s at least taking steps to deliver a secure hypervisor as well as API’s to help secure the  VM’s that run atop of it.   Where’s Citrix in this…I mean besides late and complaining they weren’t first?

So, in trade for the "open framework for security ecosystem partnership" cop-out, we get to wait for the self-perpetuating security industry hamster wheel of pain to come back full circle. 

The fact that the "industry" has "decided" that "third party vendors are required to secure any platform" simply points to the ignorance, arrogance and manifest destiny we endure at the hands of those who are responsible for the computing infrastructure we’re all held hostage with. 

Just so I understand the premise, the security industry (or is it the virtualization industry?) has decided that the security industry instead of the OS/infrastructure (virtualization) vendors are the one’s responsible to secure the infrastructure — and thus our businesses!?  What a shocker.  Way to push for change, Simon.

I can’t even describe how utterly pissed off these statements make me.

/Hoff

In my post titled "The Four Horsemen Of the Virtualization Apocalypse" I brought to light what I think are some nasty performance, resilience, configuration and capacity planning issues in regards to operationalizing virtualized security within the context of security solutions as virtual appliances/VM’s in hosts.

This point was really intended to be discussed outside of the context of virtualizing security in physical switches, and I’ll get to that point and what it means in relation to this topic in a later post.

I wanted to reiterate the point I made when describing the fourth horseman, Famine, summarized by what I called "Spinning VM straw into budgetary gold:"

By this point you probably recognize that you’re going to be deploying the same old security  software/agents to each VM and then adding at least one VA to each physical host, and probably more.  Also, you’re likely not going to do away with the hardware-based versions of these appliances on the physical networks.

That also means you’re going to be adding additional monitoring points on the network and who is going to do that?  The network team?  The security team?  The, gulp, virtual server admin team?

What does this mean?  With all this consolidation, you’re going to end up spending MORE on security in a virtualized world instead of less.

This is a really important issue because over the last few weeks, I’ve seen more and more discussions surrounding virtualization TCO and ROI calculations, but most simply do not take these points into consideration.

We talk about virtualization providing cooling, power and administrative cost-avoidance and savings.  We hear about operational efficiencies, improved service levels and agility, increased resource utilization and reduced carbon footprint. 

That’s great, but with all this virtualized and converged functionality now "simplified" into a tab or two in the management console of your favorite virtualization platform provider, the complexity and operational issues related to security have just faded into the background and been thought of as having been absorbed or abstracted away.

I suppose that might point to why many simply think that security ought to be nothing more than a drop-down menu and checkbox because in most virtualization platforms, it is!

When thinking about this, I rationalized the experience and data points against my concern related to security’s impact on performance, scale, and resiliency to arrive at what I think explains this behavior:

Most of the virtualization implementations today, regardless of whether they are client, server, production/QA or otherwise, are still internally-facing and internally-located.  There are not, based upon my briefings and research, a lot of externally-facing "classically DMZ’d" virtualized production instances.

This means that given the majority lack of segmentation of internal networks (from both a networking and security perspective,) the amount of network security controls in place are few.

Following that logic, one can then assume that short of the existing host-based controls which are put in place with every non-virtualized server install, most people continue this operational practice in their virtualized infrastructure; what they did yesterday is what they do today. 

Couple that with the lack of compelling security technologies available for deployment in the virtual hosts, most people have yet to start to implement multiple security virtual appliances on the same host.

Why would people worry about this now?   It’s not really a problem…now.

When we start to see folks ramp up virtual host-based security solutions to protect against intra-vm threats and vulnerabilities (whether internally or externally-facing) as well as to prevent jail-breaking and leapfrog attacks against the underlying hypervisors, we’ll start to see these problems bubble to the surface.

What are your thoughts?  Are you thinking about these issues as you plan your virtualization roll-outs?

/Hoff