Rational Survivability

It’s all about awareness, and as you’ll come to read, there’s just not enough of it when we talk about the security implications of virtualizing our infrastructure.  We can fix this, however, without resorting to FUD and without vendors knee-jerking us into spin battles. 

I’m attending VMworld 2007 in San Francisco, looking specifically for security-focused information in the general sessions and hands-on labs.  I attended the following sessions and a hands-on lab yesterday:

• Security Hardening and Monitoring of VMware Infrastructure 3 (Hands-on Lab)
• Security Architecture Design and Hardening VMware Infrastructure 3 (General Session)
• Using the Secure Technical Implementation Guide (STIG) with VMware Infrastructure 3 (General Session)

I had high hopes that the content and focus of these sessions would live up to the hype surrounding the statements by VMware reps at the show.   As Dennis Fisher from  SearchSecurity wrote, there are some  powerful statements coming from the VMware camp on the security of virtualized environments and how they are safer than non-virtualized deployments.  These are a bold, and in my opinion, dangerously generalized statements to make when backed up with examples which beg for context:

To help assuage customers’ fears, VMWare executives and security
engineers are going on the offensive and touting the company’s ESX
Server as a more secure alternative to traditional computing setups.
Despite the complexity of virtualized environments, they are inherently
more secure than normal one-to-one hardware and operating system
environments because of the hypervisor’s ability to enforce isolation
among the virtual machines, Mukundi Gunti, a security engineer at
VMWare said in a session on security and virtualization Tuesday.

"It’s a much more complex architecture with a lot of moving parts.
There are a lot of misconceptions about security and virtualization,"
said Jim Weingarten, senior technical alliances manager at VMWare, who
presented with Gunti. "Virtual machines are safer."

So I undertook my mission of better understanding how these statements could be substantiated empirically and attended the sessions/labs with as much of an open mind as I could given the fact that I’m a crusty old security dweeb.

Security Hardening and Monitoring of VMware Infrastructure 3 (Hands-on Lab)
The security hardening and monitoring hands-on lab was very basic and focused on general unix hardening of the underlying RH OS as well as SWATCH log monitoring.  The labs represented the absolute minima that one would expect to see performed when placing any Unix/Linux based host into a network environment.  Very little was specific to virtualization.  This  session presented absolutely nothing new or informative.

Security Architecture Design and Hardening VMware Infrastructure 3 (General Session)
The security architecture design and hardening session was at the very end of the day and was packed with attendees.  Unfortunately, it was very much a re-cap of the hands-on lab but did touch on a couple of network-centric design elements (isolating the VMotion/VIC Management ports onto separate NICs/VLANs, etc) as well as some very interesting information regarding the security of the virtual switch (vSwitch.)  More on this below, because it’s very interesting.

The Epiphany
This is when it occurred to me, that given the general roles and constituent responsibilities of the attendees, most of whom are not dedicated network or security folks, the disconnect between the "Leviathan Force" (the network and network security admins) and the "Sysadmins" (the server/VMM administrators) was little more than the mashup of a classic turf battle and differing security mindsets combined with a lack of network and information security-focused educational campaigning on the part of VMware.

After the talk, I got to spend about 30 minutes chatting with VMware’s Kirk Larsen (Engineering Product Security Officer) and Banjot Chanana (Product Manager for Platform Security) about the lack of network-centric security information in the presentations and how we could fix that.

What I suggested was that since now we see the collapse and convergence of the network and the compute stacks into the virtualizaton platforms, the operational impact of what that means to the SysAdmins and Network/Information Security folks is huge. 

The former now own the keys to the castle whilst the latter now "enjoy" the loss of visibility and operational control.  Because the network and InfoSec folks aren’t competently trained in the operation of the VM platforms and the SysAdmins aren’t competently trained in securing (holistically — from the host through to the network) them, there’s a natural tendency for conflict.

So here’s what VMware needs to do immediately:

1. Add a series of whitepapers and sessions that speak directly to assuage the fear of the virtualization unknowns targeting the network and InfoSec security staffers. 
2. Provide more detail and solicit feedback relating to the technical roadmaps that will get the network and InfoSec staffer’s visibility and control back by including them in the process, not isolating them from it.
   
3. Assign a VMware community ombudsman to provide outreach and make his/her responsibility to make folks in our industry aware — and not by soundbites that sponsors contention — that there are really some excellent security enhancements that virtualization (and specifically VMware) bring to the table.
4. Make more security acquisitions and form more partnerships.  Determina was good, but as much as we need "prevention" we need "detection" — we’ve lost visibility, so don’t ignore the basics.
   
5. Stop fighting "FUD" with "IAFNAB" (It’s a feature, not a bug) responses
6. Give the network and InfoSec folks solid information and guidance against which we can build budgets to secure the virtualization infrastructure before it’s deployed, not scrap for it after it’s already in production and hackbait.
   
7. Address the possibility of virtualization security horizon events like Blue Pill and Skoudis’ VM Jail escapes head-on and work with us to mitigate them.
8. Don’t make the mistakes Cisco does and just show pretty security architecture and strategy slides featuring roadmaps that are 80% vapor and call it a success.
9. Leverage the talent pool in the security/network space to help build great and secure products; don’t think that the only folks you have to target are the cost-conscious CIO and the SysAdmins.
10. Rebuild and earn our trust that the virtualization gravy train isn’t an end run around the last 10 years we’ve spent trying to secure our data and assets.  Get us involved.
    

I will tell you that both Kirk and Banjot recognize the need for these activities and are very passionate about solving them.  I look forward to their efforts. 

In the meantime, a lot of gaps can be closed in a very short period by disseminating some basic network and security-focused information.  For example, from the security architecture session, did you know that the vSwitch (fabric) is positioned as being more secure than a standard L2/L3 switch because it is not susceptible to the following attacks:

• Double Encapsulation
• Spanning Tree Floods
• Random Frames
• MAC Address Flooding
• 802.1q and ISL Tagging
• Multicast Brute Forcing

Basic stuff, but very interesting.  Did you know that the vSwitch doesn’t rely on ARP caches or CAM tables for switching/forwarding decisions?  Did you know that ESX features a built-in firewall (that stomps on IPtables?)  Did you know that the vSwitch has features built-in that limit MAC address flux and provides features such as traffic shaping and promiscuous mode for sniffing?

Many in the network and InfoSec career paths do not, and they should.

I’m going to keep in touch with Kirk and Banjot and help to ensure that this outreach is given the attention it deserves.  It will be good for VMware and very good for our community.  Virtualization is here to stay and we can’t afford to maintain this stare-down much longer.

I’m off to the show this morning to investigate some of the vendor’s like Catbird and Reflex and what they are doing with their efforts.

/Hoff