Rational Survivability

The resiliency and availability of security appliances in virtual environments is a focus of my "Four Horsemen of the Virtualization Security Apocalypse" presentation which I delivered during Blackhat last week.

In my session I discussed some detailed scenarios of the architectural adjustments to infrastructure that virtualizing physical security appliances require and what that means to the overall resiliency,
reliability, performance and scalability of systems which depend upon security controls in the form of physical appliances today.

Specifically, I highlighted some of the limitations of using security virtual appliances and demonstrated how relying upon virtual security appliances can actually decrease the security posture and increase risk in our environments today.

One of my examples illustrated how it may become necessary to combine multiple security virtual appliances on a cluster separate from the VM’s they are protecting in order to achieve the availability, performance, scalability, and resiliency that we get out of dedicated in-line physical appliances today.

If you prefer a simpler example, I also presented the simpler example of a firewall virtual appliance front-ending 10 production VM’s.  I like the former because it directly contrasts the physical and completely virtual.

For the sake of illustrating a point, imagine if one were actually able to satisfy business, security and compliance requirements using this virtualized security architecture in a production environment built upon the same virtualization platform as non-security guests.*

Not to pick on my friends at VMware, but today’s license timebomb issue delivered by a VMware patch is pretty nasty and sends my hackles skyward when contemplating what it would mean were one to virtualize the security infrastructure. 

Basically, this issue caused by an update rendered certain VMs unusable after the update.  If one or more VM’s on an updated host happened to be a security appliance or security control taken down for patching or rotated for re-purposing, etc. imagine your surprise post-patch.

Remember that security applications are very topology and state-sensitive and
unlike other apps. that just care about an IP address with which to spew packets
ethereally, security appliances/applications need to bind access
policies with affinity in order to protect assets behind them.  As we
all know, when something doesn’t work, we invoke the SysAdmin prime
directive: "Blame the firewall!" 😉

Events such as this may cause some to give pause to enterprise security architects before migrating security functions to virtual appliances, especially given where we are today with what I presented in terms of high-availability options within single-cluster hosts with virtual security appliances. 

In reality, it will probably cause people to consider what virtualization means as an overall contributor to operational risk for any physical system conversion — regardless of vendor — since any VA/VM would be affected, but let’s think about this as if one had virtualized the security infrastructure.

While it’s true that for guests we have DR options and snapshotting that would make roll backs an easy affair, this shines a spotlight on the difficulties with patching the underlying virtualization platform and what that means to operational resilience.

The notion of a homogeneous virtualization platform are certainly compelling; easier administration, patching, configuration, standardization, reduced costs, etc.  However, the notion of a monoculture "operating system" has its downfalls also.  This issue clearly highlights one of them.

I’m not suggesting that there are not opportunities for virtualizing certain security functions, but as I pointed out in my talk, many of the required topologies and high-availability options present in mature physical security appliances are not available in the virtual appliance world.

Today’s issue highlights the need for very careful planning when comes to what, when and if one should
virtualize security functions.

When in doubt, refer to Hoff’s Corollary:

/Hoff

* You might want to look at what a platform like the Crossbeam X-Series can give you that "normal" virtualization security platforms cannot, as it mitigates some of the issues mentioned in my talk.

** Of relevance is my blog post from back in January titled "On Patch Tuesdays For Virtualization Platforms"

Based upon feedback from attendees at Blackhat, my talk, "The Four Horsemen of the
Virtualization Security Apocalypse," went over well and I really had a lot of
fun delivering it. It’s had a TON of coverage.

Despite the positive feedback from folks, it seems the foreboding narrative of the apocalypse has carried over into the real world due to a rather unfortunate journalistic misinterpretation of the facts.

It’s only fair to state that I have been critical in the past of others in our line of work who have complained of their inability to control the output of their direct interviews with the press and analysts as misquotes and misunderstandings arise.

Perhaps this is a little karmic payback for my outspokenness, as after my talk at Blackhat, I have now enjoyed the fruits of journalistic distortion firsthand.  It’s important to note that this was not the result of a direct interview, but rather the inaccurate reporting of a reporter sitting in the audience of my talk.  I was never contacted with questions or asked for clarification or review.

Many of the points I made in my presentation were reflected upon poorly and my perspective butchered, but one specific item is causing me some serious grief in a professional capacity.  It cast a rather crappy pall on the rest of my Blackhat and Defcon experience (more on that later.)

One of the "Four Horsemen" which represents a critical issue in virtualization security is that of the hidden costs involved in virtualizing security.  The point I made, and the language I used to consistently describe it multiple times appears below:


To be perfectly clear, what I obviously said was that "virtualizing security will not save you money, it will cost you more."

What Ellen Messmer reported in her Network World article was that I said "Virtualization will not save you money, it will cost you more.”

Now, this may not seem like much of a difference, but it’s a profoundly impacting dissimilarity.

It’s a dangerous rephrase that has now caused significant pain for me that I’m going to have to deal with once I return from vacation.  It’s been picked up and re-printed/adapted so many times without validation that I can’t keep count any longer.

You see, I work as the security architect for the division of a company who is maniacally focused on designing, deploying and supporting heavily-virtualized realtime infrastructure for our customers.  One of the (obvious) value propositions of virtualization/RTI is cost savings/reduction/avoidance which I specifically referenced during my presentation as a well-established fact and reasonable motivation for virtualization.

You can probably imagine the surprise of folks when they read Ellen’s article which is written in a way that directly contradicts our corporate messaging and the value proposition offered to our clients.  It reflects rather poorly on me and my company.

And just to be clear, my scorn was not directed at the "network industry" or the "virtualization industry" as reported in the article; the context of my entire talk was the security industry, a point sorely missed.

This article reads like the output result of a bad game of "telephone."

I intend to contact Ellen Messmer and ask for a retraction as well as corrections of multiple other mistakes in the article, but as we all know, there’s no real retraction on the Internet.  All I can offer is my presentation, the video recording of it and the recollection of the 500+ others that were in the audience when I presented (including numerous other reporters.) 

The only other thing left to do is to sheepishly admit that despite the fact that this was not an interview that I or anyone else could control or influence for correctness, Joanna Rutkowska was essentially correct in her assertion during our last debate that you cannot control the press, despite best efforts. 

Even though I’ve never had a problem of this degree in the almost 15 years of doing this sort of thing, I humbly submit to her on that point.

/Hoff

I’m interrupting vacation in SoCal with the family and trucking up to Vegas for this next week’s forthcoming Blackhat and Defcon extravaganzas.  I’m getting into Vegas on Sunday, 8/3 around 4pm and leave on the 10th after Defcon.

I’ll be attending the Microsoft Ninjitsu training on Monday/Tuesday so I expect my Windows Fu will be strong as bull after the conference 😉

I’m speaking on the first day of the briefings.  My talk (Network Track – Augustus 5&6) is titled "The Four Horsemen of the Virtualization Security Apocalypse" and is from 1:45-3:00pm on August 5th.  Hope to see you there:

Despite shiny new stickers on the boxes of our favorite security vendors’ products that advertise "virtualization ready!" or the hordes of new startups emerging from stealth decrying the second coming of security, there exists the gritty failed reality of attempting to replicate complex network and security topologies in virtualized environments.

This talk will clearly demonstrate that unless we radically rethink our approach, the virtualization security apocalypse is nigh!

This talk will focus on both securing virtualization as well as virtualizing security; from virtualization-enabled chipsets to the hypervisor to the VM’s, we’ll explore the real issues that exist today as well as those that are coming that aren’t being discussed or planned for.

There are a bunch of security weenies who use Twitter who are attending one or both venues.  You can find a list of them (thanks, Zach) at the official Security Twits webspace.

See you at the show(s)

/Hoff

From AndyITGuy who summed it up perfectly:

For everything else there’s karma

• DNS vulnerability being discovered – $0
• Working with vendors to fix the problem – $0
• Having details released "accidentally" – $0
• Having exploit written by WhiteHat – $0
• Your company being pwned by your exploit – PRICELESS

Per the article above "Now he’s one of the first victims of such an attack. "It’s funny," he said. "I got owned."*

Yeah, real funny. 

/Hoff

* There’s lots of thrashing going on as to the veracity of HD’s quote rearding being owned.  Regardless of the theatrics involved, it’s interesting food for thought when the result of exploit research might be turned against the researcher…

I wanted to make you aware of a "new" excellent budding resource for VMware infrastructure, VMware’s VI:Ops – Virtual Infrastructure Operations.  Steve Chambers of VMware pointed me over to the site which is growing in both content and contributors.

VI:Ops currently includes the following sections:

• Strategies and solutions using virtualization
• Building
  and managing virtual infrastructure with open, industry standards
• Securing virtual infrastructure against risk and for compliance
• Managing and operating virtual infrastructure in the enterprise
• Automate everything virtual to be agile and efficient

Check out the site and join the community!

/Hoff

One of my responsibilities as security cruise ship entertainment director is to distill the most complex things down into bite-sized digestible nuggets of chewy informative goodness whilst ensuring a good time is had by all.

It is in this spirit that I offer this gem regarding the release of PoC/Exploit code by supposed "whitehats" immediately after the disclosure of a nasty vulnerability.  This post is random, of course, and is in no way a reference to any current event.

This quip was brought to you via Twitter which managed to stay up and functional long enough for me to tweet it:

POC code for near-zero day ‘sploits is like SPAM advertising penis-extending drugs…the only dick it’s helping is the one writing it…

That is all.

/Hoff

Update: Check it out!  Leo Laporte and Steve Gibson read my poem on their Security Now podcast.  Thanks for the radio voice, Leo!

—

A few months ago
Kaminsky discovered a flaw.
It was with DNS,
It was nasty and raw

He decided than rather
to disclose all at once
he’d instead only tell people
who’d fix it in months

So some meetings were had
and work soon began
vendors wrote patches
coordinated by Dan

Fast forward some time
out the closet it came
some researcher types
got into the game

Dan’s rules were quite simple,
that in 30 days
he’d present during Blackhat
and we’ll all be amazed

A bunch of big egos
called Dan on a bluff
said his vuln was a copy
of 10 year old stuff

So Dan swore them on handshakes
and details were provided
and those same cocky claims
soon all but subsided

It seems that Dan’s warnings
weren’t baseless at all
Said the same skeptical hackers
"the risk isn’t that small!"

So Blackhat was nearing
the web didn’t break
then out came a theory
from our friend Halvar Flake

No sooner had he posted
and described the vuln’s guts
than Matasano’s blog surfaced,
kicked the web in the nuts

It said "Halvar’s right!"
we’ll no longer keep quiet.
The post’s ripple effect
caused a nasty ‘net riot

The blog quickly was pulled
but the cat’s out of the bag
the arms race began
since there’s no longer a gag

Meanwhile the issues of honor and trust
rehashed the debate
of when disclosure goes bust

So Dan’s days of thirty
we never did see
thirteen is OK
but I issue this plea

When researchers consider
how to disclose and thus when
will you think of the users?
How it might affect them?

This ego-fueled rush
to put your name on a vuln
has a much bigger impact
than you might have known

If the point here is really
to secure and protect
then consider what image
you really project

In this case the vuln.
is now in the wild
an exploit is coming
DNS soon defiled

The arms race has started
and the clock now is ticking
If you haven’t yet patched
you’ll soon take a licking

I’m not taking sides really
on the disclosure debate
but rather the topic
of patch early or late

What good is disclosure
if the world couldn’t cope
with the resultant attacks
if we’ve all got just hope?

There’s two sides to this issue
both deserve merit
but Dan’s rep has been smeared
I say let’s just clear it

—

Happy patching everyone! ;(

/Hoff

Hat tip to Rothman for this.

I don’t know if Stiennon is off his meds or simply needed to re-post something from 2001 to meet an editorial quota, but his Network World article titled "The Most Important Networking Trend of 2008" ties thus far with the "Evolution of Dance" as my vote for most entertaining Internet content.

Richard’s epiphany goes something like this:

• Multifunction network devices that have the ability to "route" traffic and combine security capabilities are the ‘next big thing’
• If a company offers a multifunction network device that has the ability to "route" traffic and combine security capabilities but have the misfortune of using Linux as the operating system, they will "…forever be pigeon-holed as SMB solutions, not ready for enterprise
  prime time."

• The Wall Street Journal issued "… the year’s most important article on networking" in an article titled "New Routers Catch the Eyes of IT Departments" which validates the heretofore undiscovered trend of convergence and commoditization!
   
• "Real" network security players such as Cisco, Juniper and Redback are building solutions to this incredible new trend and because of the badge on the box, will be considered ready for "…enterprise prime time."
   
• The WSJ article talks about the Cisco ASR1000 router as the penultimate representation of this new breed of converged "network security" device.
   
• Strangely, Stiennon seems to have missed the fact that the operating system (IOS-XE) that the ASR1000 is based on is, um, Linux.  You know, that operating system that dictates that this poor product will "…forever be pigeon-holed as SMB solutions, not ready for enterprise
  prime time."

Oh, crap!  Somebody better tell Cisco!

So despite the fact that Cisco ASR1000 is positioned as an edge device as are these crazy solutions called UTM devices, it seems we’re all missing something because somehow a converged edge device now counts as being able to provide a "secure network fabric?"

In closing, allow me to highlight the cherry on top of Stiennon’s security sundae:   

Have you ever noticed how industry "experts" tend to get stuck in
a rut and continue to see everything through the same lens despite
major shifts in markets and technology?

Yes, Richard, I do believe I have noticed this…

Funny stuff!

/Hoff

For the second time in some months, Amazon’s S3 (Simple Storage Service,) one of the most "invisibly visible" examples of the intersection of Web2.0 and cloud computing, has suffered some noticeable availability hiccups. 

Or, if you prefer to use Amazon’s vernacular "elevated error rates" 😉

Many well-known companies such as Twitter rely upon content hosted via Amazon’s S3 which is billed as offering the following capabilities:

Amazon S3 provides a simple web services interface
that can be used to store and retrieve any amount of data, at any time,
from anywhere on the web. It gives any developer access to the same
highly scalable, reliable, fast, inexpensive data storage
infrastructure that Amazon uses to run its own global network of web
sites. The service aims to maximize benefits of scale and to pass those
benefits on to developers.

It’s not realistic to think that infrastructure as complex as this won’t suffer service disruption, but one has to wonder what companies who rely on the purported resiliency of the "cloud" from a single provider do in cases where like it’s namesake, the skies open up and the service takes a dump?

I’ll go one further.  If today you happen to use S3 for content hosting and wanted like-for-like functionality and service resiliency with a secondary provider, would your app. stack allow you to pull it off without downtime?

What happens if your apps are hosted in a cloud, too?

Sounds like a high-pressure front to me…

Next up: "CPE Security Is Dead(?): All Hail Security in the Cloud(?)"

😉

/Hoff

I was having an interesting discussion the other evening at BeanSec with Jeanna Matthews from Clarkson University.  Jeanna is one of the authors of what I think is the best book available on Xen virtualization, Running Xen.

In between rounds of libations, the topic of Hypervisor-neutral, VM portability/interoperability between the virtualization players (see right) came up.  If I remember correctly, we were discussing the announcement from Citrix regarding Project Kensho:

Santa Clara, CA » 7/15/2008 » Citrix Systems, Inc.
(Nasdaq:CTXS), the global leader in application delivery
infrastructure, today announced “Project Kensho,” which will deliver
Open Virtual Machine Format (OVF) tools that, for the first time, allow
independent software vendors (ISVs) and enterprise IT managers to
easily create hypervisor-independent, portable enterprise application
workloads. 
These tools will allow application workloads to be imported
and run across Citrix XenServer™, Microsoft Windows Server 2008 Hyper-V™ and VMware™ ESX virtual environments. 

On the surface, this sounded like a really interesting and exciting development regarding interoperability between virtualization platforms and the VMs that run on them.  Digging deeper, however, it’s not really about virtualization at all; it’s about the delivery of applications and services — almost in spite of the virtualization layer — which is something I hinted about at the end of this post.

I am of the opinion that virtualization is simply
a means to an end, a rationalized and cost-driven stepping-stone along the path of
designing, provisioning, orchestrating, deploying, and governing a more agile, real time
infrastructure to ensure secure, resilient, cost-effective and dynamic delivery of service.

You might call the evolution of virtualization and what it’s becoming cloud computing.  You might call it utility computing.  You might call it XaaS.  What many call it today is confusing, complex, proprietary and a pain in the ass to manage.

Thus, per the press release regarding Project Kensho, the notion of packaging applications/operating environments up as tasty little hypervisor-neutral nuggets in the form of standardized
virtual appliances that can run anywhere on any platform is absolutely appealing and in the long term, quite necessary.*

However, in the short term, I am left wondering if this is a problem being "solved" for ISV’s and virtualization platform providers or for customers?  Is there a business need today for this sort of solution and is the technology available to enable it?

Given the fact that my day job and paycheck currently depends upon crafting security strategies, architecture and solutions for real time infrastructure, I’m certainly motivated to discuss this.  Mortgage payment notwithstanding, here’s a doozy of a setup:

Given where we are today with the heterogeneous complexity and nightmarish management realities of our virtualized and non-virtualized infrastructure, does this really solve relevant customer problems today or simply provide maneuvering space for virtualization platform providers who see their differentiation via the hypervisor evaporating?

While the OVF framework was initially supported by a menagerie of top-shelf players in the virtualization space, it should come as no surprise that this really represents the first round in a cage match fight to the death for who wins the application/service delivery management battle.

You can see this so clearly in the acquisition strategies of VMware, Citrix and Microsoft.

Check out the remainder of the press release.  The first half had a happy threesome of Citrix, Microsoft and VMware taking a long walk on the beach.  The second half seems to suggest that someone isn’t coming upstairs for a nightcap:

Added Value for Microsoft Hyper-V

Project Kensho will also enable customers to leverage the
interoperability benefits and compatibility between long-time partners
Citrix and Microsoft to extend the Microsoft platform.  For example,
XenServer is enhanced with CIM-based management APIs to allow any
DMTF-compliant management tool to manage XenServer, including Microsoft
System Center Virtual Machine Manager. And because the tools are based
on a standards framework, customers are ensured a rich ecosystem of
options for virtualization.  In addition, because of the open-standard
format and special licensing features in OVF, customers can seamlessly
move their current virtualized workloads to either XenServer or
Hyper-V, enabling them to distribute virtual workloads to the platform
of choice while simultaneously ensuring compliance with the underlying
licensing requirements for each virtual appliance.

Project Kensho will support the vision of the Citrix Delivery Center™
product family, helping customers transform static datacenters into
dynamic “delivery centers” for the best performance, security, cost
savings and business agility. The tools developed through Project
Kensho will be easily integrated into Citrix Workflow Studio™ based
orchestrations, for example, to provide an automated, environment for
managing the import and export of applications from any major
virtualization platform.

Did you catch the subtlety there?  (Can you smell the sarcasm?)

I’ve got some really interesting examples of how this is currently shaking out in very large enterprises.  I intend to share them with you, but first I have a question:

What relevance do hypervisor-neutral virtual appliance/machine deployments have in your three year virtualization roadmaps?  Are they a must-have or nice-to-have? Do you see deploying multiple hypervisors and needing to run these virtual appliances across any and all platforms regardless of VMM?

Of course it’s a loaded question.  Would you expect anything else?

/Hoff

* There are some really interesting trade-offs to be made when deploying virtual appliances.  This is the topic of my talk at Blackhat this year titled "The Four Horsemen of the Virtualization Apocalypse"