Rational Survivability

UPDATE: Here’s the latest version of the presentation as I updated it for SecTor in Canada.  It includes many additions as well as modified single slides of the animated ones.

You can find the slides from October 2008 here.

There were some significant differences in the slides that were on the CD issued by the Blackhat folks and what I delivered.

You might be interested in them.  I’ve exported the presentation to PDF with each animation built as a separate slide – in some cases that means there are 5-6 slides with advancing bullets, graphics, etc.  As annoying as that may be, it fixes the mess of the positional overlay problem you’ll see if you view the PDF from the CD.

As an important note, my slides are designed to accompany my speaking, not the other way around, so in some cases they don’t explain themselves.  This is by design 😉

I will be giving updates to this presentation throughout the rest of the year since it’s a presentation designed to communicate the virtualization “state of the art” as it relates to VirtSec.  So, if you attend a conference and see this talk advertised, it will have new/updated content.

Be warned, this PDF is huge (~55 MB) because my slides are intensely graphical.

Enjoy.

/Hoff

(In 5 days there have been almost 1000 downloads of this preso.  If any of you have feedback, I’d really appreciate it.  Thanks)

One of my key topics in my Blackhat presentation "The Four Horsemen of the Virtualization Security Apocalypse" focused on today’s lack of state-synchronized high availability/load-balanced solutions for security virtual appliances that offer the same functionality as their physical appliance counterparts.

I used an example from a vendor I didn’t name in one of my slides to illustrate that if you attempted to replicate the options you have today in physical appliances in a virtual construct, you would not be able to (using this particular solution suite as an example.)

That vendor was Check Point and I was referring to their SPLAT-based virtual appliance. 

The version that was available for ESX environments when I created my presentation offered no load-balanced HA capabilities whatsoever as stated in the release notes. 

However, today I received the announcement of Check Point’s VPN-1 Virtual Edition (virtual appliance.)

Based upon what I am reading in the administrator’s guide, VE now includes support for ClusterXL HA FW/VPN state-synchronized load sharing across one or more ESX hosts.

This is a great first step in the right direction.

We should expect more solutions to start arriving shortly to address many of the issues I brought up as the current "state of the art," and this is a good example of that. 

There are numerous caveats and limitations, many of which I spoke directly about in my presentation with many of them related to the interdependencies of network topologies, virtual networking and interaction with VMware’s integrated HA/clustering solutions…which was one of the other key topics in my talk 😉 I’ll update some of them as an addendum to this post shortly. Updated below.

You can find information regarding Check Point’s VPN-1 Virtual Edition here.

/Hoff

Updated: If you read the release notes, you’ll notice these little gems.  I don’t have the time to go through each of these limitations, but they’re significant and go to the point that all things are not equal in V versus P:

• Interface bonding on the virtual machine running the VPN-1 Virtual Edition is not supported
  with ClusterXL.
• VMotion is not supported
• VPN-1 gateways in the Bridge Mode must have their internal and external interfaces connected
  to port groups that are configured in promiscuous mode.
• VPN-1 gateways in the Bridge Mode are not supported with ClusterXL.
• Virtual machines may be connected to a maximum of four different virtual switches. This may
  limit the number of virtual networks protected by a VPN-1 Virtual Edition machine. This
  limitation can be overcome using VLANs.

I’ll give them props for this one, though, given its refreshing honesty:

• VPN-1 Virtual Edition does not protect the VMkernel.

I just responded to a comment from Iben Rodriguez on one of my virtualization and PCI blog entries from a while back and posted an observation while at the same time managed to make a funny (see the title.)

I wanted to both reflect upon Iben’s comment as well as chuckle a bit.

From what I extracted from his comment, Iben is suggesting that perhaps virtualization should not affect an auditor’s approach or differentiate the audit process from a physical server depending upon the definition of a "server:"

Is an ESX Host a server?

It should be considered similar to the chassis holding a bunch of blade servers. 

These have management ports on separate networks, with LDAP authentication over security protocols like ssh and ssl.

And why not treat them as a hybrid device with different network switches, storage controllers, etc?

Vmware has recently removed the word "Server" from after the ESX product name…

It’s not a server, it’s a hypervisor.

It’s not a server, it’s a switch.

By defining what a server is and is not a PCI Audit should be pretty straight forward.

I think this is a messy question and one we ought to continue to address.  I need to go and check out my ISACA references to seek guidance on this matter from a, um, "higher" source 😉 I do think that ultimately this is a very subjective issue, to which I responded:

The answers to your questions/suppositions are quite simple:

"It all depends upon the auditor."

Most of the folks I’ve spoken to recently are essentially counting
upon the ignorance of the auditors and the general confusion regarding
terminology and technology to glide by at this point.

Server/blade/hypervisor/switch … it’s all fun and games until someone loses a (PC)I… 😉

"As long as I put in place the same host controls I do in a physical
environment and not tell the auditor it’s virtualized, it’s all good
and what they don’t know, won’t hurt me."

Sad but true.

The resiliency and availability of security appliances in virtual environments is a focus of my "Four Horsemen of the Virtualization Security Apocalypse" presentation which I delivered during Blackhat last week.

In my session I discussed some detailed scenarios of the architectural adjustments to infrastructure that virtualizing physical security appliances require and what that means to the overall resiliency,
reliability, performance and scalability of systems which depend upon security controls in the form of physical appliances today.

Specifically, I highlighted some of the limitations of using security virtual appliances and demonstrated how relying upon virtual security appliances can actually decrease the security posture and increase risk in our environments today.

One of my examples illustrated how it may become necessary to combine multiple security virtual appliances on a cluster separate from the VM’s they are protecting in order to achieve the availability, performance, scalability, and resiliency that we get out of dedicated in-line physical appliances today.

If you prefer a simpler example, I also presented the simpler example of a firewall virtual appliance front-ending 10 production VM’s.  I like the former because it directly contrasts the physical and completely virtual.

For the sake of illustrating a point, imagine if one were actually able to satisfy business, security and compliance requirements using this virtualized security architecture in a production environment built upon the same virtualization platform as non-security guests.*

Not to pick on my friends at VMware, but today’s license timebomb issue delivered by a VMware patch is pretty nasty and sends my hackles skyward when contemplating what it would mean were one to virtualize the security infrastructure. 

Basically, this issue caused by an update rendered certain VMs unusable after the update.  If one or more VM’s on an updated host happened to be a security appliance or security control taken down for patching or rotated for re-purposing, etc. imagine your surprise post-patch.

Remember that security applications are very topology and state-sensitive and
unlike other apps. that just care about an IP address with which to spew packets
ethereally, security appliances/applications need to bind access
policies with affinity in order to protect assets behind them.  As we
all know, when something doesn’t work, we invoke the SysAdmin prime
directive: "Blame the firewall!" 😉

Events such as this may cause some to give pause to enterprise security architects before migrating security functions to virtual appliances, especially given where we are today with what I presented in terms of high-availability options within single-cluster hosts with virtual security appliances. 

In reality, it will probably cause people to consider what virtualization means as an overall contributor to operational risk for any physical system conversion — regardless of vendor — since any VA/VM would be affected, but let’s think about this as if one had virtualized the security infrastructure.

While it’s true that for guests we have DR options and snapshotting that would make roll backs an easy affair, this shines a spotlight on the difficulties with patching the underlying virtualization platform and what that means to operational resilience.

The notion of a homogeneous virtualization platform are certainly compelling; easier administration, patching, configuration, standardization, reduced costs, etc.  However, the notion of a monoculture "operating system" has its downfalls also.  This issue clearly highlights one of them.

I’m not suggesting that there are not opportunities for virtualizing certain security functions, but as I pointed out in my talk, many of the required topologies and high-availability options present in mature physical security appliances are not available in the virtual appliance world.

Today’s issue highlights the need for very careful planning when comes to what, when and if one should
virtualize security functions.

When in doubt, refer to Hoff’s Corollary:

/Hoff

* You might want to look at what a platform like the Crossbeam X-Series can give you that "normal" virtualization security platforms cannot, as it mitigates some of the issues mentioned in my talk.

** Of relevance is my blog post from back in January titled "On Patch Tuesdays For Virtualization Platforms"

I wanted to make you aware of a "new" excellent budding resource for VMware infrastructure, VMware’s VI:Ops – Virtual Infrastructure Operations.  Steve Chambers of VMware pointed me over to the site which is growing in both content and contributors.

VI:Ops currently includes the following sections:

• Strategies and solutions using virtualization
• Building
  and managing virtual infrastructure with open, industry standards
• Securing virtual infrastructure against risk and for compliance
• Managing and operating virtual infrastructure in the enterprise
• Automate everything virtual to be agile and efficient

Check out the site and join the community!

/Hoff

I was having an interesting discussion the other evening at BeanSec with Jeanna Matthews from Clarkson University.  Jeanna is one of the authors of what I think is the best book available on Xen virtualization, Running Xen.

In between rounds of libations, the topic of Hypervisor-neutral, VM portability/interoperability between the virtualization players (see right) came up.  If I remember correctly, we were discussing the announcement from Citrix regarding Project Kensho:

Santa Clara, CA » 7/15/2008 » Citrix Systems, Inc.
(Nasdaq:CTXS), the global leader in application delivery
infrastructure, today announced “Project Kensho,” which will deliver
Open Virtual Machine Format (OVF) tools that, for the first time, allow
independent software vendors (ISVs) and enterprise IT managers to
easily create hypervisor-independent, portable enterprise application
workloads. 
These tools will allow application workloads to be imported
and run across Citrix XenServer™, Microsoft Windows Server 2008 Hyper-V™ and VMware™ ESX virtual environments. 

On the surface, this sounded like a really interesting and exciting development regarding interoperability between virtualization platforms and the VMs that run on them.  Digging deeper, however, it’s not really about virtualization at all; it’s about the delivery of applications and services — almost in spite of the virtualization layer — which is something I hinted about at the end of this post.

I am of the opinion that virtualization is simply
a means to an end, a rationalized and cost-driven stepping-stone along the path of
designing, provisioning, orchestrating, deploying, and governing a more agile, real time
infrastructure to ensure secure, resilient, cost-effective and dynamic delivery of service.

You might call the evolution of virtualization and what it’s becoming cloud computing.  You might call it utility computing.  You might call it XaaS.  What many call it today is confusing, complex, proprietary and a pain in the ass to manage.

Thus, per the press release regarding Project Kensho, the notion of packaging applications/operating environments up as tasty little hypervisor-neutral nuggets in the form of standardized
virtual appliances that can run anywhere on any platform is absolutely appealing and in the long term, quite necessary.*

However, in the short term, I am left wondering if this is a problem being "solved" for ISV’s and virtualization platform providers or for customers?  Is there a business need today for this sort of solution and is the technology available to enable it?

Given the fact that my day job and paycheck currently depends upon crafting security strategies, architecture and solutions for real time infrastructure, I’m certainly motivated to discuss this.  Mortgage payment notwithstanding, here’s a doozy of a setup:

Given where we are today with the heterogeneous complexity and nightmarish management realities of our virtualized and non-virtualized infrastructure, does this really solve relevant customer problems today or simply provide maneuvering space for virtualization platform providers who see their differentiation via the hypervisor evaporating?

While the OVF framework was initially supported by a menagerie of top-shelf players in the virtualization space, it should come as no surprise that this really represents the first round in a cage match fight to the death for who wins the application/service delivery management battle.

You can see this so clearly in the acquisition strategies of VMware, Citrix and Microsoft.

Check out the remainder of the press release.  The first half had a happy threesome of Citrix, Microsoft and VMware taking a long walk on the beach.  The second half seems to suggest that someone isn’t coming upstairs for a nightcap:

Added Value for Microsoft Hyper-V

Project Kensho will also enable customers to leverage the
interoperability benefits and compatibility between long-time partners
Citrix and Microsoft to extend the Microsoft platform.  For example,
XenServer is enhanced with CIM-based management APIs to allow any
DMTF-compliant management tool to manage XenServer, including Microsoft
System Center Virtual Machine Manager. And because the tools are based
on a standards framework, customers are ensured a rich ecosystem of
options for virtualization.  In addition, because of the open-standard
format and special licensing features in OVF, customers can seamlessly
move their current virtualized workloads to either XenServer or
Hyper-V, enabling them to distribute virtual workloads to the platform
of choice while simultaneously ensuring compliance with the underlying
licensing requirements for each virtual appliance.

Project Kensho will support the vision of the Citrix Delivery Center™
product family, helping customers transform static datacenters into
dynamic “delivery centers” for the best performance, security, cost
savings and business agility. The tools developed through Project
Kensho will be easily integrated into Citrix Workflow Studio™ based
orchestrations, for example, to provide an automated, environment for
managing the import and export of applications from any major
virtualization platform.

Did you catch the subtlety there?  (Can you smell the sarcasm?)

I’ve got some really interesting examples of how this is currently shaking out in very large enterprises.  I intend to share them with you, but first I have a question:

What relevance do hypervisor-neutral virtual appliance/machine deployments have in your three year virtualization roadmaps?  Are they a must-have or nice-to-have? Do you see deploying multiple hypervisors and needing to run these virtual appliances across any and all platforms regardless of VMM?

Of course it’s a loaded question.  Would you expect anything else?

/Hoff

* There are some really interesting trade-offs to be made when deploying virtual appliances.  This is the topic of my talk at Blackhat this year titled "The Four Horsemen of the Virtualization Apocalypse"

Edward Haletky wrote an interesting piece recently titled "CISecurity Guide to VMware Security Falls Far Short" in which he lays down some well-articulated criticisms of the first CIS benchmark for VMware.

Edward’s primary problem with the benchmark can be summarized well by this paragraph:

While the Benchmark was the first of its kind, it is nothing more than the Linux benchmark with some small changes for VMware ESX. Following these steps will increase security but it is by no means a panacea. Do not let it give you a false sense of security.

I think Edward set his expectations a little high prior to review, as I’m pretty sure the word panacea wasn’t used in the syllabus 😉

I don’t disagree with Edward that the flavor of the benchmark is very much a generic set of guidelines focused primarily on securing the underlying Linux-based service console and basic configuration for overall "system" hardening, but we need to realize a couple of things to keep the benchmark in perspective:

1. The benchmark was the first of its kind.  It’s almost 10 months old!  The second version is underway right now as a matter of fact.
2. In between when the benchmark was released and now, we’ve seen the emergence of the embedded version of VMware and much needs to change to address that.
3. The benchmark was designed to be generic and give virtual system administrators a baseline on basic security hardening, not serve as the end-all, be-all for some mythical security end-state.
4. The challenge for those of us who contributed (as I did) was that we had to keep the document vendor/tool agnostic which makes it difficult to frame solutions.
5. Lots of things have changed.

Keep in mind that this is a "level 1" benchmark whose settings/actions are as follows:

• Can be understood and performed by system administrators with any level of security knowledge and experience;
• Are unlikely to cause an interruption of service to the operating system or the applications that run on it; and
• Can be automatically monitored either by CIS Scoring Tools or by CIS Certified tools available from security software vendors. 

This isn’t about being defensive regarding the benchmark as I’ll agree that we could have done much, much more in terms of providing more meatier substance as it relates to how to better secure the ecosystem of mechanicals that a virtualized environment touches. 

However, the scope of a document that effectively addresses the security concerns across this immense landscape would be a huge undertaking.

One of the other difficulties in creating a guideline like this is the fact that those responsible for securing virtualized environments are not security professionals.  As I’ve spoken about previously, the operational realities of who is managing and securing our virtualized infrastructure is cause for concern.

Thus, when creating a guide like this, it’s best to start with the underlying basics and then branch out from there; involve the network and security teams as required.  As Edward himself wrote in this piece, "Good virtual security requires better IT teamwork," to properly secure your virtualized infrastructure, it’s going to take cooperation and expertise from many camps.    

Edward also has written a book titled "VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers."  Interestingly, I found the security sections weak for many of the same high-level reasons he listed in his review of the CIS benchmark.  Security is most definitely in the eye of the "bookholder." 😉

In the meantime, if you’re interested in some additional security/hardening guides and tools for VMware environments, check out the following:

• VMWare – Updated VI3 Security Hardening Guide (7/8/08)
• DoD/DISA – Security Technical Implementation Guide (STIG)
• NSA SNAC – VMware ESX Server 3 Configuration Guide
• Tripwire – ConfigCheck

/Hoff

Alessandro from virtualization.info and I were chatting today regarding VMware’s latest best-practices document titled "DMZ Virtualization with VMware Infrastructure." 

This is a nine page overview that does a reasonably good job of laying out many of the architectural/topological options available when thinking about taking the steps toward virtualizing what some consider the "final frontier" in the proving grounds of production-level virtualization — the (Internet-facing) DMZ.

The whitepaper was timely because I was just finishing up my presentation for Blackhat and was busy creating a similar set of high-level architectural examples to use in my presentation.  I decided to reference those in the document because they quite elegantly represent the starting points that many folks would use as a stepping off point in their virtual DMZ adventures.

…and I think it will be an adventure punctuated perhaps by evolutionary steps as documented in the options presented in the whitepaper.

As I read through the document, I had to remind myself of the fact that this was intended to be a high-level document and not designed to cover the hairy edges of network and security design. 

The whitepaper highlighted some of the reasonable trade-off’s in complexity, resiliency, management, functionality, operational expertise, and cost but given where my head and focus are today, I have to admit that it still gnawed at me from a security perspective which is still too weak for my liking.

I’ve hinted at why in my original Four Horsemen slide, and I’m going to be speaking for 75 minutes on the topic at Blackhat, so come get your VirtSec boogie on there for a full explanation…

Alessandro got dinged in a comment on his blog for a statement in which he suggested that partially-collapsed as well as fully-collapsed DMZ’s with virtual separation of trust zones "…should be avoided at all costs because they imply the inviolability of the hypervisor (at any level: from the virtual networking to the kernel) something that nor VMware neither any other virtualization vendor can grant."

This appears contradictory to his initial assessment of DMZ virtualization wherein he stated that "…there [is] nothing bad in virtualizing the DMZ as long as we are fully aware of the risks."  In a way, I think I understand exactly where Alessandro is coming from, even if I don’t completely agree with him (or at least I partially do…)

This really paints an altogether unfortunate and yet accurate picture of the circular arguments folks engage in when they combine the following topics in a single argument:

• Securing virtualization
• Virtualizing security
• Security via virtualization

In the same way that we trust our operating system vendors who provide us with the operational underpinnings of our datacenters with the hope that they will approach a reasonable level of "security" in their products, we are basically at the same point with our virtualization (OS) platform providers.

Hope is not a strategy, but it seems we’ve at least accepted it for the time being… ;(

Sure there are new attack vectors and operational risks, but the slippery slope of not being able to really quantify whether you are more or less at risk based solely on the one-dimensional data point of the infallibility of the hypervisor  and then write the whole concept off seems a little odd to me.

If you’re truly assessing risk in the potential virtualization of your DMZ, you’ll take the operational/architectural guidelines as well as the subjective business impacts into consideration.  Simply stating that one should or should not virtualize a DMZ without a holistic approach is myopic.

To circle back on the topic, the choice of whether to — and how to — virtualize your DMZ  is really starting to gain traction.  I think the whitepaper took a decent first-pass stab at exploring how one might approach it, but the devil’s in the details — or at least the devil’s 4 horsemen are 😉

/Hoff

Today was the deadline for submission for all selected Blackhat presentations. 

I’m giving a 75 minute talk titled "The Four Horsemen of the Virtualization Apocalypse" which is based upon my original blog posting here.

I dutifully uploaded my presentation to Ping and the gang at Blackhat HQ today (on time, that’s a first!) with a sigh of relief and accomplishment.  I’ve done hundreds of presentations over the years, but this one is special.

I have to say that I poured my heart and soul into this presentation.  I went all "Zen and the Art of Presentation" for most of it and I think that combined with the dozens of hours I put into the content, the diagrams and animations turned out purdy. 😉

Once BH is done, I’ll be posting it online with my narrative as I have my other presentations.

This cathartic little post is just the final little exhale of this project prior to numerous advance rehearsals, the first of which I will be inflicting upon my unwitting guests (75+ of them thus far) at my July 5th Pig Roast & Mojito festival in honor of another notch in the annual belt I’ve managed to stay alive on this hunk o’ rock.

Speaking of which, if you’re in the MA area and want an amazing cuban or southern-style pulled pork feast with all the trimmings, drop me a line as everyone’s welcome…many of the BeanSec’rs are coming, you should too!

Happy 4th/5th!

/Hoff

Thanks to Alan Shimel and his pre-Blackhat Security Bloggers Network commentary, a bunch of interesting folks are commenting on the topic of virtualization security (VirtSec) which is the focus of my preso at Blackhat this year.

Mike Rothman did his part this morning by writing up a thought-provoking piece opining on the lack of a near-term market for VirtSec solutions:

So I’m not going to talk about technical stuff. Yet, I do feel compelled to draw the conclusion that despite the dangers, it doesn’t matter. All the folks that are trying to make VirtSec into a market are basically just pushing on a rope.

That’s right. Now matter how hard you push (or how many blog postings you write), you are not going to make VirtSec into a market for at least 2 years. And that is being pretty optimistic. So for all those VCs that are thinking they’ve jumped onto the next big security opportunity, I hope your partnership will allow you to be patient.

Again, it’s not because the risks of virtualization aren’t real. If guys like Hoff and Thomas say they are, then I tend to believe them. But Mr. Market doesn’t care what smart guys say. Mr. Market cares about budget cycles and priorities and political affiliations, and none of these lead me to believe that VirtSec revenues are going to accelerate anytime soon.

Firstly, almost all markets take a couple of years to fully develop and mature and VirtSec is no different.  Nobody said that VirtSec will violate the laws of physics, but it’s also a very hot topic and consumers/adopters are recognizing that security is a piece of the puzzle that is missing.

In many cases this is because virtualization platform providers have simply marketed virtualization as being "as secure" or "more secure" than than their physical counterparts.  This, combined with the rapid adoption of virtualization, has caused a knee jerk reactive reaction.

By the way, this is completely par for the course in our industry.  If you act surprised, you deserve an Emmy 😉

Secondly, and most importantly to me, Mike did me a bit of a disservice by intimating that my pushing the issues regarding VirtSec are focused solely on the technical.  Sadly, that’s so far off base from my "fair and balanced" perspective on the matter because along with the technical issues, I constantly drum home the following:

• The biggest challenge we have with virtualization in the short term (the next couple of years — the same timeframe Mike suggests VirtSec will take to percolate) is operational and organizational, and not technical.

  You can find recent postings on that topic in posts such as Security Pros Say VirtSec Is An Operations Problem? and The Challenge of Virtualization Security: Organizational and Operational, NOT Technical   

• Along with the organizational and operational issues comes the need for visibility and visualization of what is going on within these virtual environments.

  You can find my latest entry on this (posted early today, actually) in my entry titled Visualization Through Virtualization…

"Nobody Puts Baby In the Corner"

Painting only one of the legs of the stool as my sole argument isn’t accurate and doesn’t portray what I have been talking about for some time — and agree with Mike about — that these challenges are more than one-dimensional.

The reality is that Mike is right — the budget, priority and politics will bracket VirtSec’s adoption, but only if you think of VirtSec as a technical problem.

Is VirtSec a market?  My opinion: it’s an instantiation of technology, practice and operational adjustment brought forth as a derivative of a disruptive technology and prevailing market conditions. 

Does that mean it’s a feature as opposed to a market?  No.  In my opinion, it’s an evolution of an existing market, rife with existing solutions and punctuated by emerging ones.

The next stop is how "security" will evolve from VirtSec to CloudSec…

/Hoff