Rational Survivability

I’m off to Munich for the rest of this week to keynote day two of Troopers08, hosted by my friend Enno Rey and the team at ERNW.

My talk is titled "Virtualization: Floor Wax, Dessert Topping and the End of Information Security As We Know It."

I’m sure I’m going to get hassled because I didn’t finish my VRRP fuzzing parameters for SPIKE before the weenies @ ERNW did (OK, I have an excuse — I didn’t even start) but it’s bound to be a great conference and a good time.

I got this email from Enno yesterday.  He’s German and thus obviously quite serious about this:

For those interested:

a) there will be a 10K (kilometers) run in the morning of 04/23 and 04/24, at 7 AM each. no competing here, just get some fresh air (planned time: 60 minutes). We’ve not yet figured out the exact route, given it’s airport area there shouldn’t be too many hills or stuff.
If you want to run on 04/25 or have a "double round" one of the days, pls drop me personal note.

b) the hotel seems to have a decent gym. We asked them to have it open 24h during the con and they confirmed this.

The friggin’ beer capital of the Universe and he wants us to run 10Km in the morning.

Yeah, right.

I’m looking for a local Brazilian Jiu Jitsu acacdemy, however…

Catch you all on the flipside…so long as the German customs officers don’t realize that MacOS X comes with NMAP which we all *know* is a hacking tool…<gulp!>

/Hoff

I had another post-RSA press release show up in my mailbox today from IBM again pitching their "…breakthrough research initiative from IBM X-Force and IBM Research,
code-named "Phantom", which offers businesses a new means of securing
virtualized server environments."

Besides the rumblings at RSA, I haven’t been briefed on this as of yet, but let’s explore what we have thus far, keeping it mind that this is described as an "initiative" and not a "product:"

At Phantom’s core is industry-leading network and host intrusion protection used to guard the virtual environment and the machines from the inside out. The new technology sits in a secure, isolated partition and integrates with the hypervisor – the layer of management software that coordinates calls between operating systems and computer hardware.

In this description, Phantom is confusingly framed more as a product/solution rather than an initiative and it gets a little fuzzy as to how this qualifies as integration with the hypervisor besides just sitting on top of it, but perhaps this is one of the secrets-in-stealth that defines the breakthroughs mentioned above or perhaps sadly yet another unfortunate translation from Klingon?

If one were to take a quick first-pass, it sounds like they’ve taken their software-based IBM/ISS IPS solution and turned it into a virtual appliance (that would be the "secure, isolated partition") that runs alongside the VM’s in a physical host?  This is basically what every other vendor on the planet is currently doing.  Integration with SiteProtector and interaction with the hardware-based physical appliances would make sense, too.

Playing futurist, in terms of the more broadly-reaching "initiative" angle, it might leverage some of the research IBM has already done on their secure hypervisor (sHype) or more appropriately rHype (which I believe is Xen-based) as well as the many other virtualization efforts they’ve hatched to date.

If IBM were going to commercialize this into productized offerings, besides supporting their own hypervisor(s) and virtualization platforms/operating systems first, I’d guess they would aim for supporting VMware first since that’s where the dollars are.  Or not.

IBM’s Phantom initiative aims to create virtualization security technology to efficiently monitor and disrupt malicious communications between virtual machines without being compromised. 

In addition, full visibility of virtual hardware resources would allow Phantom to monitor the execution state of virtual machines, protecting them against both known and unknown threats before they occur.

Roger.  Protect intra-vm traffic.  And because they can protect "…against both known and unknown threats before they occur" it’s psychic to boot! 😉

It is also designed to increase the security posture of the hypervisor – a critical point of vulnerability; because once an attacker gains control of the hypervisor, they gain control of all of machines running on the virtualized platform. For the first time, the hypervisor, the gateway to the virtualized world and all that lays above it, can be locked down.

I’m interested in this part because as most vendor’s pitches go, when one digs down deeper, what this really means is that *today* if one can control traffic between the VM’s which transit the vSwitch, one can potentially prevent a compromise of a VM leading to a launchpad for an attack on the hypervisor.

What’s confusing here is that despite the fact that most hypervisor platform providers consciously limit what is exposed (even in an abstracted state) by the hypervisor, vendors continue to insist that they are "integrated" with and will "lock down" the hypervisor itself.  We saw that in the dissection of the Catbird "HyperVisorShield" announcement I wrote about earlier.

Protecting the hypervisor today is really a by-product of protecting the VM’s.

Here’s another extract from additional coverage of Phantom:

Phantom is a joint effort between IBM’s X-Force threat analysis team and the company’s research division. It aims to lock down the hypervisor software that IBM systems use to manage virtual machines. "What we’re doing through Phantom is we’re implementing an IPS (intrusion prevention system)– an IPS that sits at the hypervisor layer," said Kris Lovejoy, director of strategy for IBM corporate security.
…
The researchers are also building tools that can lock down the hypervisor itself, Lovejoy added. "The hypervisor layer was built for optimum performance, not necessarily effective security," she said. "Our customers are just looking for assurance that their virtualized infrastructure is not going to be the single point of failure."

Aha!  See vendors in their press releases continue to reference THE hypervisor in a singular, monolithic manner that seems to imply that their solutions will protect and lockdown any and all hypervisors.  I know this point may not be lost on all people, but it’s become very difficult to figure out what many of these VirtSec products actually do and which platforms they support.

I think this last paragraph really intimates that in this case we’re talking about IBM’s hypervisor(s) — perhaps based upon sHype/rHype or other IBM virtualization platforms — at least at first.

I’m not knocking IBM or doubting their efforts as they’ve been at the virtualization game a long time and with the acquisition of ISS, they got a bunch of good talent and a decent product base.  I *am* just weary of claims that seem to apply research and "initiatives" in such broad strokes that it becomes difficult to sort the wheat from the chaff.

Looking forward to learning more about Phantom.

/Hoff

What was the biggest thing at RSA this year?

Information Centricity?  Been there, done that.
Security Innovation?  SO last Tuesday.
DLP?  Nope.
NAC? Nah-Uh.
GRC? Not so much.

The biggest thing at RSA this year was, of course, my conference badge:

Yo!  BeanSec! is once again upon us.  Wednesday, April 16th, 2008.

BeanSec! is an informal meetup of information security
professionals, researchers and academics in the Greater Boston area
that meets the third Wednesday of each month. 

I say again, BeanSec! is hosted the third Wednesday of every month.  Add it to your calendar.

Come get your grub on.  Lots of good people show up.  Really.

Unlike other meetings, you will not be expected to pay dues, “join
up”, present a zero-day exploit, or defend your dissertation to attend.
Map to the Enormous Room in Cambridge.

Enormous Room: 567 Mass Ave, Cambridge 02139.  Look for the Elephant
on the left door next to the Central Kitchen entrance.  Come upstairs.
We sit on the left hand side…

Don’t worry about being "late" because most people just show up when
they can.  6:30 is a good time to aim for.  We’ll try and save you a
seat.  There is a parking garage across the street and 1 block down or
you can try the streets (or take the T)

In case you’re wondering, we’re getting about 30-40 people on
average per BeanSec!  Weld, 0Day and I have been at this for just over
a year and without actually *doing* anything, it’s turned out swell.

We’ve had some really interesting people of note attend lately (I’m
not going to tell you who…you’ll just have to come and find out.)  At
around 9:00pm or so, the DJ shows up…as do the rather nice looking
people from the Cambridge area, so if that’s your scene, you can geek
out first and then get your thang on.

The food selection is basically high-end finger-food appetizers and
the drinks are really good; an attentive staff and eclectic clientèle
make the joint fun for people watching.  I’ll generally annoy you into
participating somehow, even if it’s just fetching napkins. 😉

See you there.

/Hoff

[For those of you directed here for my Blackhat 2008 presentation of the same name, the slides will be posted with a narrative shortly.  This was the post that was the impetus for my preso.

If you’d like to see a "mini-me" video version of the presentation given right after my talk, check it out from Dark Reading here.  You’ll also notice this is quite different than Ellen Messmer’s version of what I presented…]

—

I’ve written and re-written this post about 10 times, trying to make it simpler and more concise.  It stems from my initial post on the matter of performance implications in virtualized security environments here.

After a convo with Ptacek today discussing the same for a related article he’s writing, I think I’ve been able to boil it down to somewhere near its essence. It’s still complex and unwieldy, but it’s the best I can do for now.

Short of the notions I’ve discussed previously regarding instantiating the vSwitches into hardware and loading physical servers with accelerators and offloaders for security functions, there aren’t a lot of people talking about this impending set of challenges or the solutions in the short or long term.

This should be cause for alarm.

These issues are nasty.  Combined with the organizational issues of who actually owns and manages "security" in the virtualized context, this stuff makes me want to curl up in a fetal position.

So here they are, the nasty little surprises awaiting us all carried forth by the four horsemen of the virtualization security apocalypse named conquest, war, famine and death:

• Virtualized Security Screws the Capacity Planning Pooch (Conquest)
• The Network Is the Compu…oh, crap.  Never mind, it’s broken. (Death)
• Episode 7: Revenge of the UTM.  Behold the vUTM! (War)
• Spinning VM straw into budgetary gold (Famine)

In order to ameliorate these shortcomings, we’re going to have to see some seriously different approaches and rapid acceleration of solution roadmaps.  There are some startups as well as established players all jockeying to solve one or more of these problems, but they’re not going to tell you about them because, quite frankly, they are difficult to describe and may cause TPOW syndrome (Temporary Purchase Order Withholding.)

So here they are in all their splendor.  The gifts of the four horsemen, just in time to pour salt in your virtualized wounds:

1. Virtualized Security Screws the Capacity Planning Pooch (Conquest)
   If we look at today’s most common implementation methodologies for deploying security in a virtualized environment, we end up recognizing that it comes down to two fundamental approaches: (a) install software/agents from the usual suspects in the VM’s or (b) deploy security functions as virtual appliances (VA) within the physical host.

   If we look at measuring performance overhead due to option (a) I wager we’d all have a reasonably easy time of measuring and calculating what the performance hit would be.  Further, monitoring is accomplished with the tools we have today. This is a per-VM impact that can be modeled across physical hosts and in response to overall system load. No real problem here.

   Now, if we look at option (b) which is the choice of almost all emerging solutions in the VirtSec space, the first horseman’s steed just took a crap on Main street. 

   For example, let’s say that we have one (or more — see #2 and #3 below) monolithic security VA whose job it is is to secure all traffic to and from external sources to any VM in the physical host as well as all intra-VM traffic.

   You see the problem, right?  Setting aside the notion of how much memory/CPU to allocate to the VA so as not to drop packets due to overload, capacity planning completely depends upon the traffic levels, the number of VM’s on the system (which can be dynamic,) the way the virtual and physical networks are configured (also dynamic) as well as the efficiency of the software/OS combo in the VA.  Lest we forget access to system buses, hardware and the tax that comes with virtualizing these solutions.

   The very real chance exists of either overrunning the VA and dropping packets which will lead to retransmissions, etc. or simply losing valuable landscape to add VM’s because the "extra" CPU/memory you thought you had is now allocated to the security VA…

   Measuring security VA performance is a crapshoot, too.  Sure there’s VMMark, but methinks that we already have enough crap floating about in how vendors measure performance of physical appliances whose resources they control.  Can you imagine the first marketing campaigns that are sure to be launched on the first 10Gb/s virtual appliance…Oh my.

2. The Network Is the Compu…oh, crap.  Never mind, it’s broken. (Death)
   Virtualization offers some fantastic benefits, not the least of which is the capability to provide for resilience and on-demand scalability/high-availability.  If a physical server is overloaded, one might automagically allow the VMotion of critical VM’s to other lighter-loaded physical hosts.  If a process/application/VM fails on one host, spin it back up somewhere else.  Great stuff.

   Except, we’ve got a real problem when we try to apply this dynamic portability to security applications running in VA’s.  Security applications are incredibly topology sensitive. For the most part, they expect the network configuration to remain static – interfaces, VLAN’s, MAC addies, routes, IP addresses of protected nodes, etc.  If you go moving security VA’s around, they may no longer be inline with the assets they protect!

   Further, the policies that define the ACL’s that govern the disposition of traffic also don’t grok.

   But wait, there’s more!

   Replicating certain operating conditions within a virtualized environment is going to be tricky when the VirtServer admins don’t have any idea of what VRRP and multicast MAC addies are (that the security applications depend upon) and how that might affect load balancing firewall cluster members within the same physical host.  Mutliwhat?

   An example might be that you want to implement high availability load balancing for a "cluster" of firewall VA’s within a single physical host so that you don’t have to VMotion an entire server’s worth of VM’s over to another if the security VA which is inline fails (we can address HA/LB across two physical hosts later.)  It’s going to be really interesting trying to replicate in a virtualized construct what we’ve spent years gluing together in the physical world: vSwitch behavior, port groups, NIC teaming, etc.

   Lastly, I’m skipping ahead a little and treading on issue #3 below, but if one were to deploy multiple security VA’s within a single physical host to provide the desired functionality across protected VM’s, how does one ensure that traffic flow is appropriately delivered to the correct VA’s at the correct time with the correct disposition reflected up and downstream?

   There are some really difficult challenges to overcome when
   attempting to "combine" security functions in-line with one another.
   In fact, this concept is what gave birth to UTM — combining multiple
   security functions into a single platform and optimize both the control
   effectiveness, simplify management and reduce cost.

   Most UTM vendors on the market either write their own security
   stacks and integrate them, take open source code and/or OEM additional
   technologies to present what is marketed as a single "engine" against
   which traffic is cracked once and inspected based upon intelligent
   classification.  Let’s just take that at face value…and with a
   healthy grain of salt.

   My last company, Crossbeam, took a different approach.  Crossbeam
   provides a network and (security) application virtualization platform (the
   X-Series security
   service switch) and allows an operator to combine a number of discrete
   third party ISV security solutions in software in specific serialized
   and parallelized processing order based upon policy. You pick the
   firewall, IPS, AV, AS, URL filter, WAF, etc. of your choosing and
   virtualize those combinations of functions across your network as a
   service layer.

   This is the same model I am trying to illustrate in the case of server virtualization with security VA’s except that the Crossbeam example utilizes an external proprietary chassis solution.

   Here’s an overly-simplified illustration of four security
   applications as deployed within an X-series: an IPS, IDS, firewall, web
   application firewall (WAF).  These applications are instantiated once
   in the system and virtualized across the network segments connected to
   them governed by policy:

   
   Note for the purpose of simplicity I’m showing a flow path from ingress
   to egress that is symmetrical.

   Technically, egress flows could
   actually take a different path through other software stacks which
   makes the notion of "state" and how you define it (via the "network" or
   the "application") pretty darn important.  I’m also leaving out the
   complexity of VLAN configurations in this example.

   What’s interesting here is that each of these applications can often
   be configured from a network perspective as a layer 2 or layer 3
   "device," so how the networking is configured and expects to be
   presented with traffic, act on it, and potentially pass it on is really
   important.  Ensuring that flows and state are appropriately directed to
   the correct security function and is presented in the correct "format"
   with low latency and high throughput is much easier said than done.

   Can you imagine trying to do this in a virtualized instance on a server across multiple security VA’s?  There’s really no control plane to effect this, no telemetry, and the vSwitch isn’t really designed as a fabric to provide much more than layer 2 connectivity.

   Fun for the entire family!  Kid tested, virtualization approved!

3. Episode 7: Revenge of the UTM.  Behold the vUTM! (War)
   "The farce is strong with this one…"  OK, so this is a dandy.  The models today that talk about VA installations position the deployment of a single security vendor’s VA solution.  What that means is combined with the issues raised in points (1) and (2) above, we’re sort of expecting to not embrace the best-of-breed approach and instead of deploying a CHKP firewall VA, an ISS IDP VA, a McAfee Anti-malware VA, etc., we’ll just deploy a single vendor’s monolithic security stack to service the entire physical host?

   Does this model sound familiar?  See #2 above.  Well, either you’re going to do that and realize that your security ultimately sucks harder than a Dyson or you’re going to do the nasty and start to deploy multiple vendor’s security VA’s in the same physical host.

   See the problem there?  Horseman #3 reminds you of the points already raised above.  You’re going to be adding security VA’s which takes away the capacity to add valuable VM’s dynamically:
   

   
   …and then you’re going to have to deal with the issues in #2 above.  Or, you’ll just settle for "good enough" and deploy what amounts to a single UTM VA and be done with it.  Until it runs out of steam or you get your butt handed to you on a plate when you’re pwned.

   You could plumb in a Crossbeam or even less complex single-vendor appliance solutions, but then you’re going to find yourself playing ping-pong with traffic in and out of each VM, through the physical NICs, and in/out of the appliances.  Latency is going to kill you.  Saturation of the pipe is going to kill you.  Your virtual server admin is going to kill you, especially since he won’t have the foggiest idea of what the hell you’re going on about.

   Further, if you’re thinking VMsafe’s going to save you trouble in either #2 or #3, it ain’t.  VMsafe sets its hooks on a per VM basis and then redirects to a VA/VM within each physical host.  It’s settings in the first release are quite coarse and you can’t make API calls outside of the physical hosts, so the "redirects" to external appliances won’t work.  Even if they did, there’s no control plane to deal with the "serialization" I demonstrate above.

4. Spinning VM straw into budgetary gold (Famine)
   By this point you probably recognize that you’re going to be deploying the same old security  software/agents to each VM and then adding at least one VA to each physical host, and probably more.  Also, you’re likely not going to do away with the hardware-based versions of these appliances on the physical networks.

   That also means you’re going to be adding additional monitoring points on the network and who is going to do that?  The network team?  The security team?  The, gulp, virtual server admin team?

   What does this mean?  With all this consolidation, you’re going to end up spending MORE on security in a virtualized world instead of less.

There is lots of effort going on to try to force-fit entire existing markets of solutions in order to squeeze a little more life out of investments made in products, but expect some serious pain in the short term because you’re going to be dealing with all of this for the next couple of years for sure.

I hope this has opened your eyes to some of the challenges we’re going to face moving forward.

Finally, let us solemnly remember that:

Joanna Rutkowska took the time to respond to my "open letter" that I wrote this weekend regarding her presentation at RSA.  I truly appreciate that.  It was a little barbed, but so was mine, but all’s fair in love and blogging.

I chortled, however, when I realized that I was deserved of a response only for the following reasons:

1) technorati.com reported the blog’s authority as above 100 which suggests it has a reasonable number of readers, and also

2) because I believe this is a good example of the social engineering techniques used by my opponents

I just about coughed my latte through my nose when I read that.

Just to be clear, Joanna, I’m not an "opponent" and despite your assertions, I don’t provide PR services for anyone.  I *do* however rather like the fact that you’ve anointed me with the madly-133t-skillz of a social engineer. 😉

Let me make it perfectly clear (because I don’t think I have) that I find your research incredibly interesting and your work compelling.  What I question is the relevancy across use cases and the way in which you choose to present it.  This is despite your bemoaning to the contrary, the way in which you surrender your words to the fates (i.e. the press) and seem powerless to be able to ensure what you said is printed in context accurately. 

Rather than continue the enthralling debate regarding the vagaries of municipal fire codes, let me get to the meat of the redress which is what I focused on in the first place: what you said and what you may have meant to say are two different things, Joanna.

To wit:

2. Type I vs. Type II hypervisors confusion.

Hoff then switches to the actual content of the presentation and writes this:
“When I spoke to you at the end of your presentation and made sure that I understood correctly that you were referring specifically to type-2 hosted virtualization on specific Intel and AMD chipsets, you conceded that this was the case.”

This simply is an incorrect statement! On the contrary, when describing the security implications of nested virtualization (which was the actual new thing I was presenting at the RSA), I explicitly gave an example of how this could be used to compromise type I hypervisors. Kindly refer to slides 85-90 of my presentation that can be downloaded here.

I said that the code we posted on bluepillproject.org indeed targets type II hypervisors and the only reason for that being that it has been built on top of our New Blue Pill code that was designed as a Windows kernel driver.

This is exactly why I and a couple of other folks came up to speak with you at the end of your talk.  It was not at all clear as to which case you were referring.  I humbly accept the responsibility for a lack of cognition here.  When I sought that clarification, you specifically answered as I mention above which confirmed my understanding.  To that end, the gentleman behind me responded "Yeah, that’s what I wanted to ask, too" and thanked you for the clarification.  Now you’re suggesting that what we heard was not what you said…

3. Shit not giving. Mr. Hoff goes even further:

“When I attempted to suggest that while really interesting and intriguing, your presentation was not only confusing to many people but also excluded somewhere north of 80% of how most adopters have deployed virtualization (type-1 "bare-metal" vs. type-2 hosted) as well as excluding the market-leading virtualization platform, your response (and I quote directly) was: I don’t give a shit, I’m a researcher.”

Now that was a hard blow! I understand that the usage of such a slang expression by an Eastern European female during an informal conversation with a native speaker must have made an impression on him! However, I couldn’t give such an answer to this very question, simply because of the reasons given in point #2 (see above).

I don’t care whether you’re an "Eastern European female" or a cross-dressing circus clown from Bolivia.  What does concern me is that first you suggest that your making that statement must have been shocking to me and then you immediately maintain you didn’t say it…and you throw in the gender card!  Nice.

Joanna, your dismissal using this exact phrasing is exactly what got me riled up.  Your dishonesty and/or confusion about what you said and what you think you said is the entire point you’re missing…except hysterically you claim you are a victim of the very issue I highlight:

So, then Hoff quotes the Forbes article that was written after my presentation and accuses me that the article (written by some Forbes reporter) was too sensationalist. I definitely agree the article was very sensationalist (but correct) and when I saw the article I even got angry and even wanted to write a blog about it (but as the article was actually correct, I had no good arguments to use against it).

And you know why I was so angry? Because I actually spent over 40 minutes with this very Forbes reporter in the RSA’s speaker’s lounge just after my speech, I spent that time on clarifying to that guy what my presentation was about and what it was not about and what was the main message of the presentation. Still, the reporter had his own vision of how to write about it (i.e. make it into a sensation) and I hardly, as it turned out, could do anything about it…

Perhaps the fault is ours, but perhaps you should accept some of the responsibility here, too?  If you continue to be misunderstood, misquoted, and misrepresented, perhaps it has something more to do with than the fact that your intellect is "…too technical for an average CISSP to understand it?"  Perhaps you are hard to understand?  Perhaps you don’t do a good job of explaining?  Perhaps the language gap is confusing things?

Look, I find the following assertion really interesting, and had you allowed me to ask the question, would have loved to have discussed it with you further:

"Keep[ing] hypervisors simple, do not put drivers there, as otherwise we would get to the same point where we are with current OSes these days, i.e. no kernel security at all!”

…but I didn’t get a chance to.  I actually resonate with your assertion.  I didn’t bring it up because that’s not what I had a problem with.

Finally, to your closing point:

Now I wonder, maybe Christofer Hoff doesn’t do PR for any VMM vendor, maybe he just didn’t listen carefully to my presentation. Maybe he’s just one of those many guys who always know in advance what they want to hear and selectively pick up only those facts that match their state of mind? Otherwise, why would he not realize that my presentation was actually a pro-virtualization one and needed no (false) counter-arguments?

Sigh.

I came to your presentation the way I do to every other I attend.  With an open mind, open ears and a closed mouth.  I listened carefully, was confused by what I thought were contradictory statements between your slides and what you were saying and sought clarification.  Upon clarification and subsequent condescending dismissal, I closed my mouth and my ears and formed my conclusion based upon your response.

Perhaps you’ll use this as an opportunity to reflect upon how you present and interact with people.  Perhaps you won’t.  I know I will.  Either way, I appreciate your research and your response to my "letter."

/Hoff

The intellectual integrity scandal of the century has reared its ugly head once again. 

At RSA in the bar of the Westin, I was confronted by an unruly mob of Ex-@Stakers, fueled by their infamous ringleader Dan "El Guapo" Geer, who cornered me rather forcefully between a Bellini and a half-empty bottle of Dos Equis.  He suggested that were I not to cooperate, a true demonstration of punctuated equilibrium would be at my expense.

It was during this mental waterboarding session that I was unduly pressured to provide a public admission of guilt and forced to yield to photographic evidence of the event after El Guapo craftily scratched out "my " confession on a bar napkin which read "Hoff stole my preso."  At least he spelled my name correctly.

This was a sad day, indeed.  El Guapo sank my battleship 🙁

/Hoff

I’m going to highlight a prediction I had on a forthcoming security
offering from yet-to-be-named security solution providers for
virtualized environments as well as something I overheard at RSA.

In the next few days, I’m going to be releasing my post on the
evolution of some really concerning performance and configuration
limitations of security solutions in virtualized environments and this
will make a lot more sense, but until then, grok this…

Here’s Item #1 – Return of the Big, Honkin’ NIC Card…

Remember back when 3Com released this little beauty?

3Com® 10/100 Secure Server NIC
 

Server IPSec and 3DES Encryption at Wire Speeds
 

The 3Com® 10/100 Secure Server NIC is custom-designed for servers that
demand high performance and end-to-end security. Its onboard security
processor works with Windows 2000 or XP to offload key processing
tasks, reducing the load imposed on the CPU.

It never really took off and has long since been discontinued, but
here’s where I reckon we’re going to see a rebirth (like bellbottoms)
of something similar from security vendors, either as a NIC or an
offload card sitting in the virtual host.

In a virtualized server, most of the emerging security solutions are
going to take the form of agents/applications running in VM’s or as
virtual appliances in the host.  This is all going to be run in
software, with limitations on memory, CPU and I/O.  Imagine every flow
whether inter-host or intra-VM having to bounce back and forth across
the vSwitch and the security functions in software.

Ugh.

Despite API’s like VMsafe, which allow for hooks on a per-VM basis to
"redirect" traffic to a VM/VA for disposition in software, imagine if
instead of just having IPSec on a NIC, we also had DPI, firewall, IDP,
AV and other security functions also.

Rather than doing all of this stuff in software, the
agents/applications or virtual appliances could offload or allow the
hardware to perform them on their behalf.  This could take the form of
FPGA’s or custom silicon like Cavium’s multi-core Octeon security
processors.

This is where the argument of "hey, all we need is COTS multicore hardware to scale" simply falls apart.

It’s not at all an original idea, as we’ve had offload/acceleration cards in appliances/’servers for a long time, but when the performance and
configuration limitations of virtual hosts arise, I predict we’ll see these things crop
up as a "solution" that is "new." 😉

Here’s item #2 – Bait and (Virtual) Switch

I’ve talked previously about virtualization platform providers like VMware ultimately providing a way of modularizing/isolating the vSwitch functionality in the VMM and allowing third parties to instantiate their own vSwitch instead. 

Further, I’ve written about how I/O virtualization is likely to change the way and where the virtual networking is performed. 

Intel is rumored (was this news at RSA, I can’t tell?) to be taking
another approach which is that they intend to embed the vSwitch
functionality directly into the underlying CPU chipsets.  This makes the vSwitch not so much ‘v’ (virtual) any longer.
You’ll have the network switching fabric and functions in the CPU itself.

I’m sure that if Intel is considering this, then AMD would not be far behind.

Thus some version of an upcoming CPU would provide this capability
natively, interfacing with the NIC card (or the super NIC above) and
the VMM.  This brings up some really interesting questions, no?

More later.

/Hoff

Dear Joanna:

I attended your session at the RSA conference last week titled "Security Challenges in Virtualized Environments" and was compelled to write you given our debate which you ended somewhat abruptly at the conclusion of your presentation.

Before I start in on the meat of the topic, I’m going to do what you seem to continue NOT to do.  Specifically, I am going to make clear certain disclosures and frame the context of this note in a way that I hope everyone can understand.

Sadly, there will not be an accompanying eight-slide melange of virtual machine state transitions, mention of TLB misses, GIF0 emulation or ASID conflicts…

Back to your presentation.

As the room filled to over capacity before your talk began, you were upset and couldn’t seem to understand why the conference organizers would not let people spill over from seats and sit on the floor and in the aisles to hear you speak.  The fact that fire and safety codes prohibit packing a room beyond capacity was something you attributed to people being "…crazy in America."  Go figure.

So let me further raise your ire by introducing you to another crazy American rule of law that is somewhat related: we don’t think it’s a good idea to yell "fire!" in a crowded theater, either.

What does this have to do with your presentation?  It’s quite simple actually.   I think that the way in which you are presenting your research is intentionally designed to be sensational first and concise and accurately portrayed a distant last.

During your presentation at RSA and throughout other presentations,
you have illustrated how your research featuring Blue Pill technology affects hardware-based type-2 hosted virtualized environments rather than type-1 bare-metal installs.

In many cases, given the depth and complexity of your presentations,
less experienced audience members and members of the press have
completely confused or overlooked this distinction and left your presentation thinking that your research and your testing applies directly and unequivocally to both environments, despite the fact that you continue to highlight Microsoft’s Vista desktop operating system as your test case.

When I spoke to you at the end of your presentation and made sure
that I understood correctly that you were referring specifically to type-2 hosted virtualization on specific Intel and AMD chipsets, you conceded that this was the case.

When I attempted to suggest that while really interesting and intriguing, your presentation was not only confusing to many people but also excluded somewhere north of 80% of how most adopters have deployed virtualization (type-1 "bare-metal" vs. type-2 hosted) as well as excluding the market-leading virtualization platform, your response (and I quote directly) was:

"I don’t give a shit, I’m a researcher."

So my problem with that answer is three-fold Joanna:

• As a researcher who is also actively courting publicity for commercial gain and speaking at
  conferences like RSA which are less technical and more "executive" in
  nature, you have a responsibility to clarify and not obfuscate
  (intentionally or otherwise) the facts surrounding your research.
  Allowing the continued sensationalized coverage of your research
  without clarification is not allowing concerned people to make clearly
  informed decisions regarding risk.

• No less than five times during your presentation, you highlighted marketing material in the form of graphics from Phoenix, positioned their upcoming products and announced/credited both Phoenix and AMD as funding your research. 

  Further, there have been announcements suggesting that Phoenix is looking to commercialize Blue Pill not as a rootkit but as an "ultra-thin" hypervisor.  This makes it hard to decide where the breakpoint between your "research" versus their "commercial" begins.

• Continuing to openly and negatively disparage those who seek to challenge your assertions is unprofessional.  Certainly you can disagree with them, but regardless of their approach or attitude, the continued pejorative nature of your rebuttals is getting stale. 

I think it’s only fair to point out that given your performance, you’re not only an "independent researcher" but more so an "independent contractor."  Using the "I’m a researcher" excuse doesn’t cut it.

I know it’s subtle and lots of folks are funded by third parties, but they also do a much better job of drawing the line than you do.

Despite your position on the matter and unlike you, I do give a shit, Joanna.  I care very much that your research as presented to the press and at conferences like RSA isn’t only built to be understood by highly skilled technicians or researchers because the continued thrashing that they generate without recourse is doing more harm than good, quite frankly.

Now, I know you can’t control the press or what they print, but you certainly don’t seem to invest much in terms of ensuring accuracy or clarifying the corner cases you’re talking about.  Here’s an example from a Forbes article based upon your RSA presentation:

At the security industry’s big annual confab, the RSA Conference, going on this week in San Francisco, security researcher Joanna Rutkowska described a new type of virtualization-based malware that could be used to take control of a machine running virtualization software. Because virtualization allows companies to store many virtualized software "images" of computers on a single physical machine, an attack like the one Rutkowska envisions would allow a hacker not only to control a single machine but to siphon data from any virtual machine it contains.

Rutkowska, the founder of security research firm Invisible Things Lab, in Warsaw, Poland, isn’t the first to target virtualization as a weak point in the emerging IT landscape. In the past few months, security researchers have revealed bugs in practically every piece of virtualization software, including products from virtualization heavyweights VMware (nyse: VMW – news – people ) and Microsoft (nasdaq: MSFT – news – people ).

Exploiting those bugs, attackers can use what researchers call "virtual machine escape," or "hyperjacking." By taking control of the hypervisor, the piece of software that controls all the virtual computers within a machine, an attacker can "escape" from any single virtual computer hosted on the machine and quickly multiply his or her access to a company’s data.

But the attack Rutkowska outlined goes even further: she described how an intruder could install what she calls a "blue pill," a second, malicious hypervisor that controls the original hypervisor and all of the virtual machines beneath it. Examining any PC or server hosted on the machine, it would appear that the machines were hosted normally by a hypervisor, but, she argues, it would be tough to detect another hidden hypervisor intercepting data or manipulating the virtualized computers.

"When you use virtualization to build malware, there are no hooks, nothing you can see within an operating system," she says.

So this reporter walked away from your presentation and basically represents — like every other reporter I have seen — that every virtualization platform is covered under your research and is susceptible to attack regardless of chipset, operating system or application, a fact that you already conceded during our short exchange as not being the case.  Do you not see how this can be confusing?

In this scenario, I can personally attest that Fortune 100 companies deploying VMware ESX 3i are unable to determine whether they are at risk or not.  You could certainly take the low-road and blame this on those interpreting your presentations this way, or perhaps recognize that this could be a direct result of your efforts.

Despite the fascinating research, I’m really disappointed in how you choose to continue to allow inaccurate representation of your research to continue unabated.  Instead of inflammatory, sensational and inaccurate portrayals, you could instead be really helping to educate the world in a way not dependent on fear, uncertainty and doubt.

I look forward to your next presentation.  I just hope it’s more accurately tempered next time so as not to cause the figurative stampede from the theater when there’s actually no fire.

/Hoff

Update: Joanna responded here.  I retorted playing ping to her pong here.  Enjoy.

It seems I forgot to specifically call out the fact that I’ll be in San Francisco attending the RSA Conference this week.

Monday had me at the America’s Growth Capital conference with the remainder of the week spread between sessions, briefings, meeting up with old friends and making new ones.  I’m leaving back to Boston Friday morning.

I’m speaking on Wednesday (DEPL-201)

/Hoff