Rational Survivability

It’s no doubt apparent that trafficking in the ideas and concepts surrounding both virtualized security and securing virtualized environments really honks my horn.  I’ve been writing about it a lot lately, and it’s starting to garner some very interesting amounts of attention from lots of different sources.

One of those sources sent me an email after reading some of my ramblings and framed a discussion point that I was writing about anyway, so I thought it a perfect opportunity to discuss it.

Specifically, when a disruptive emerging technology bursts onto the scene with many of the threats and vulnerabilities associated with said technology being mostly theoretical, conceptual and virtual in nature, how does one have a very real conversation with management regarding what should be done proactively to (and please forgive me both ISS and ISS-naysayers) "get ahead of the threat."

That is, how do you start talking about the need to assess and make actionable, if possible, the things necessary to secure such an impacting technology?  Asked not to be identified when I quoted him, I believe one of my readers summed this up quite nicely:

"I really enjoy your blog posts about virtualization security, since it’s a challenge I’m dealing with right now. The real problem I’m finding is explaining the security issues to people who don’t get security in general, and double-don’t-get-it in the context of virtualization.

The two points I really try to get across are:

1. the fact that there aren’t any common, well-known attacks specific to virtualization in the wild (guest hopping etc) is not a good thing, it’s a BAD thing; they’re coming!

2. a virtual server is like a little mini-network where essentially none of our existing security measures apply (I guess I’m mostly thinking of IDS here)

Am I hitting the right points, do you think? Where else can I go with this, since the "threat" is pretty much "I don’t know but something someday?"

My response is straightforward.   I think that he’s dead-on inasmuch as explaining virtualization and the risks associated with it is difficult, mostly because the "threats" are today mostly theoretical and the surface area for attack — or the vulnerabilities for that matter — just aren’t perceived as being there.

So the normal thing to do is just suggest that what we have will be applied to solve the "same" problems in the virtualized context and we’ll deal with the virtualization-specific threats and vulnerabilities when they become more "real." <sigh>

We can shout to the treetops about what is coming, but people don’t generally invest in security proactively because in many cases we’ve seemed to accept that the war is lost and we’re just looking to win a battle every once and a while.  <sigh^2>

It doesn’t help that we’re trying to build business cases to start thinking about investing in securing virtualized environments when the threats and vulnerabilities are so esoteric and by manner of omission executives are basically told that security is something they do not need to focus on any differently in their virtualization deployments.

So I only have a few suggestions for now:

1. I’d use my preso. to help lubricate the conversation a little; it sums most of this up nicely
2. Don’t make the mistake of suggesting the sky is falling — it may be, but that’s not going to get you timeshare or share of wallet
3. In this nascent market, we have to communicate the potential exposure and elevated risk in
   the language of and terms associated with business; why should you spend time and money on this versus, say, patch management.
4. You better have an answer to this one: "Virtualization is going to save us money, now you want to spend more to secure it!?"
5. Abstract the discussion related to investment in terms of pushing vendors in your portfolio (by spending time/money) on making sure they will have something to offer you when you need it and start assessing your business and IT plans to see how they align to policy today
6. Start to build what will be the best practices for what your virtualized environments ought to look like with what you know now, BEFORE you start having to put them into production next week
7. Talk with your auditors — make them your allies.  Ask them how they expect to audit and assess your virtual environments (be careful what you ask for, however)
8. Use what you have; you’re going to have to for a while anyway.
9. Start testing now; demonstrate empirically how existing compensating controls will/will not satisfy your security policies in a virtualized construct
10. Keep calm.  By the time we get around to cleaning this mess up, we’ll have another pile right around the corner.  This is a continuum, remember?  Same crap, different decade. At least we have twitter and facebook now.

In closing, and without sounding like a clucking chicken, check out this summary of a recent vulnerability disclosure on how to run arbitrary code on a VMware GuestOS thanks to a "feature" in VMware’s scripting automation API. Dennis Fisher over at SearchSecurity did a nice write-up about Mark Burnett’s recent discovery:

The folks at VMware have been in the news quite a bit of late,
thanks to their big IPO and their discreet acquisition of Determina a
couple of weeks ago. Now, the company’s core virtualization product is
getting some attention, but not the kind company executives will like.
Mark Burnett, an independent security consultant and author, recently
posted a long description of a vulnerability in VMware’s scripting automation API that he found.

The vulnerability comes down to this: The API allows any script on
the host machine to execute code and take other actions on any virtual
machine that’s running on the PC, without requiring any credentials on
the guest operating system. This presents a number of problems, as
Burnett points out:

The problem is that a malicious script running within
the context of a regular user on my desktop can run administrator-level
scripts on any guest I am currently logged in to. Using Ctrl+Alt+Del to
lock the desktop of those machines does not prevent VIX from executing
commands on the guest. Even if I log out of each guest machine the
malware can just queue the command to run the next time I log in at the
console of the guest OS.

However, this is in fact a feature that the VMware developers
intentionally included. VMware told Burnett that, in essence, anyone
who can access the virtual machine APIs on a machine can access the
virtual hard disks anyway and would be able to attack the PC from that
direction. But it seems to me that Burnett is on to something here.
Sure, there are plenty of other methods for attacking virtual machines,
but that doesn’t mean this should be ignored.

Burnett also has found a way to mitigate the problem by adding a switch to the VMX config file.

So Mogull is back on the bench and I’m glad to see him blogging again. 

As I type this, I’m listening to James Blunt’s  new single "1973" which is unfortunately where Rich’s timing seems to be on this topic.  ‘Salright though.  Can’t blame him.  He’s been out scouting the minors for a while, so being late to practice is nothing to be too wound up about.

<If you can’t tell, I’m being sarcastic.  I only wish that Rich was
when he told me that his favorite TexMex place in his hometown is
called the "Pink Taco."  That’s all I’m going to say about that…>

The notion of the HyperJackStack (Hypervisor Jacking & Stacking) is actually a problem set that has been discussed at length and in the continuum of these discussions happened quite a while ago. 

To put it bluntly, I believe the discussion — for right or wrong — stepped over this naughty little topic months ago in lieu of working from the bottom up for the purpose exposing fundamental architectural deficiencies (or at least their potential) in the core of virtualization technology.  This is an argument that parallels dissecting a BLT sandwich…you’re approaching getting to the center of a symmetric stack so which end you start at is almost irrelevant.

The good/bad VMM/HV problem has really been relegated to push-pin on the to-do board of all of the virtualization vendors and this particular problem has been framed by said vendors to be apparently solved first operationally from the management plane and THEN dealt with from the security perspective.

So Rich argues that after boning up on Joanna and Thom’s research that they’re arguing the wrong case completely for the dangers of virtualized rootkits.  Instead of worrying about undetectability of this or that — pills and poultry be damned — one should be focused on establishing the relative disposition of *any* VMM/Hypervisor running in/on a host:

Problem is, they’re looking at the wrong problem. I will easily concede
that detecting virtualization is always possible, but that’s not the
real problem. Long-term virtualization will be normal, not an
exception, so detecting if you’re virtualized won’t buy you anything.
The bigger problem is detecting a malicious hypervisor, either the main
hypervisor or maybe some wacky new malicious hypervisor layered on top
of the trusted hypervisor.

To Rich’s credit, I think that this is a huge problem and one that deserves to be solved.  That does not mean that I think one is the "right" versus "wrong" problem to solve, however.  Nor does it mean this hasn’t been discussed.  I’ve talked about it many times already.  Maybe not as eloquently…

The flexibility of virtualization is what provides the surface expansion of vectors for threat; you can spin up, move or kill a VM across an enterprise with a point-click.  So the first thing to do before trying to determine if a VMM/HV is malicious is to detect its presence and layering in the first place…this is where Thom/Joanna’s research really does make sense.

You’re approaching this from a different direction, is all.

Thom responded here, and I have to agree with his overall posture; the notion of putting hooks into the VMM/HV to allow for "external" detection mechanisms for the sake solely of VMM/HV rootkit detection is unlikely given the threat, but we are already witness to the engineered capacities to allow for "plug-ins" such as Blue Lane’s that function "along side" the HV/VMM and there’s nothing saying one couldn’t adapt a similar function for this sort of detection (and/or prevention) as a value-add.

Ultimately though, I think that the point of response boils down to the definition of the mechanisms used in the detection of a malicious VMM/HV.  I ask you Rich, please define a "malicious" VMM/HV from one steeped in goodness. 

This sounds like in practice, it will come down to yet another iteration of the signature-driven IPS circle jerk to fingerprint/profile disposition.  We’ll no doubt see anomaly and behavioral analysis used here, and then we’ll have hashing, memory firewalls, etc…it’s going to be the Hamster Wheel all over again.  For the same reason we have trouble with validating security and compliance state for anything more than the cursory checks @ 30K feet today, you’ll face the same issue with virtualization — only worse.

I’ve got one for you…how about escaping from the entire VM "jail" entirely…Ed Skoudis over @ IntelGuardians just did an interview with the PaulDotCom boys on this topic…

I believe one must start from the bottom and work up; they’re trying to make up for the fact that this stuff wasn’t properly thought through in this iteration and are trying to expose the issue now. In fact, look at what Intel just announced today with vPro:

New in this product is Intel Trusted Execution Technology (Intel
TXT, formerly codenamed LaGrande). Intel TXT protects data within
virtualized computing environments, an important feature as IT managers
are considering the adoption of new virtualization-enabled computer
uses. Used in conjunction with a new generation of the company’s
virtualization technology – Intel Virtualization Technology for
Directed I/O – Intel TXT ensures that virtual machine monitors are less
vulnerable to attacks that cannot be detected by today’s conventional
software-security solutions. By isolating assigned memory through this
hardware-based protection, it keeps data in each virtual partition
protected from unauthorized access from software in another partition.

So no, Ptacek and Joanna aren’t fighting the "wrong" battle, they’re just fighting one that garners much more attention, notoriety, and terms like "HyperJackStack" than the one you’re singling out.  😉

/Hoff

P.S. Please invest in a better setup for your blog…I can’t trackback to you (you need Halo or something) and your comment system requires registration…bah!  Those G-Boys have you programmed… 😉

On September 24-25th, InfoWorld will host their Virtualization Executive Forum in NYC which promises "…two days of
technical breakout sessions, case studies and industry expertise on
server, desktop, application, storage and file virtualization
technologies."

Here’s the overview:

Designed for those
who are evaluating where to begin and for those already implementing
virtualization technologies, InfoWorld’s Virtualization Executive Forum
features:

• Analyst perspectives on innovative uses of virtualization adoption
  rates and trends, and policy-based datacenter automation
  
• In-depth
  sessions examining Virtual Machines and Security, Open Source
  Virtualization, Business Continuity/Disaster Recovery, and more.
• Industry
  Keynotes from IT end users addressing the challenges, pitfalls,
  results, and benefits of their implementations
• A
  spotlight on Green IT practices and its potential for cost savings and
  reducing power and cooling needs in large datacenters.
  

In
addition to the in-depth case studies and industry panels you have come
to expect from InfoWorld’s Executive Forums, this fourth edition has
added another key ingredient to the mix: more opportunities for you and your peers to  collaborate and share experiences.

For an "executive forum" they have an interesting split-track breakout agenda; one track features case studies and the other focuses on technical presentations and panels.

Here’s the rub, did you notice that the word "security" appears only twice in the entire agenda, once in the keynote address and once more in a case-study breakout session on day two regarding applications of virtualization.  While I recognize that this is supposedly targeted at "executives," let’s take a look at the technical track breakout topics:

• 
  Vendor Crossfire: x86 Server Virtualization
• Getting Started with Server Virtualization
• Technical Track: Physical to Virtual Migration
• Leveraging Virtualization for Information Availability and Business Continuity
• Lessons from Big Iron: The Power of RISC UNIX Virtualization
• Open Source Hypervisor: Zeroing in on Xen
• VM Management and Monitoring
• Scaling Virtual Infrastructure

Not a mention of security in the bunch.  This is asinine. If you’re at all curious as to why security is an after-thought in emerging markets, look no further than this sort of behavior. 

…and don’t just tell me that security is "assumed."

If the executives who attend this two day forum walk away with a head full of fun new ideas and cautionary tales regarding virtualization and the closest thing to security they got was the valet guarding the doughnuts during the break, don’t anybody get surprised in 18 months when the house of cards come tumbling down.

InfoWorld, what the hell!?  How about ONE session — even a panel — titled something as simple as "Virtualization and Security – A Discussion You Need to Have."

In fact, you’re welcome to at least just print out my presentation from a couple of days ago and give it to your attendees.  At least they’ll walk away with something relating to security and virtualization.  850+ people from my blog already have more information on security and virtualization *for free* than is being presented at the forum.

Listen, I feel so strongly about this that I’ll speak for free on the topic — I’ll pay my own hotel, airfare, etc…and you can keep the doughnuts during the break.

By the way, I find it deliciously ironic that when I clicked on the "Visit Virtualization Portal" link in the above graphic, I was greeted by this little gem:

I’m sure this is probably running on a "real" server.  A virtualized instance would never have this sort of problem, right? 😉

/Hoff

This fifth episode of Take5 interviews Allwyn Sequeira, SVP of Product Operations for Blue Lane.  

First a little background on the victim:

Mr.
Palmer is the President and Chief Executive Officer of Blue Lane
Technologies. Previously, Jeff served as President of GetThere, the
leading online corporate travel procurement solution, based in Menlo
Park, CA. While at the company, GetThere grew to become the largest
factor in the corporate online travel market, holding 70% market share
among the top 100 US corporations, #1 share in the overall market with
more than 3000 customers worldwide, and transaction volumes exceeding
one million per month. GetThere went public on the NASDAQ in 1999 and
was acquired by Sabre Holdings (NYSE: TSG) for $750 million in 2000.
         
Earlier in his career, Mr. Palmer served in a variety of general
management and marketing officer roles at Tristrata Security, Memco
Software, Pilot Software and BBN, establishing experience in enterprise
networking, security, and applications software.
         
Mr. Palmer holds a Bachelor of Science degree from the Massachusetts
Institute of Technology and an MBA from Harvard Business School.

Allwyn Sequeira is Senior Vice President of Product Operations at Blue
Lane Technologies, responsible for managing the overall product life
cycle, from concept through research, development and test, to delivery
and support. He was previously the Senior Vice President of Technology
and Operations at netVmg, an intelligent route control company acquired
by InterNap in 2003, where he was responsible for the architecture,
development and deployment of the industry-leading flow control
platform. Prior to netVmg, he was founder, Chief Technology Officer and
Executive Vice President of Products and Operations at First Virtual
Corporation (FVC), a multi-service networking company that had a
successful IPO in 1998. Prior to FVC, he was Director of the Network
Management Business Unit at Ungermann-Bass, the first independent local
area network company. Mr. Sequeira has previously served as a Director
on the boards of FVC and netVmg.

Mr. Sequeira started his career as a software developer at HP in the
Information Networks Division, working on the development of TCP/IP
protocols. During the early 1980’s, he worked on the CSNET project, an
early realization of the Internet concept. Mr. Sequeira is a recognized
expert in data networking, with twenty five years of experience in the
industry, and has been a featured speaker at industry leading forums
like Networld+Interop, Next Generation Networks, ISP Con and RSA
Conference.

Mr. Sequeira holds a Bachelor of Technology degree in Computer
Science from the Indian Institute of Technology, Bombay, and a Master
of Science in Computer Science from the University of Wisconsin,
Madison.   

Allwyn, despite all this good schoolin’ forgot to send me a picture, so he gets what he deserves 😉
(Ed: Yes, those of you quick enough were smart enough to detect that the previous picture was of Brad Pitt and not Allwyn.  I apologize for the unnecessary froth-factor.)

 Questions:

1) Blue Lane has two distinct product lines, VirtualShield and PatchPoint.  The former is a software-based solution which provides protection for VMware Infrastructure 3 virtual servers as an ESX VM plug-in whilst the latter offers a network appliance-based solution for physical servers.  How are these products different than either virtual switch IPS’ like Virtual Iron or in-line network-based IPS’s?

IPS technologies have been charged with the incredible mission of trying to protect everything from anything.  Overall they’ve done well, considering how much the perimeter of the network has changed and how sophisticated hackers have become. Much of their core technology, however, was relevant and useful when hackers could be easily identified by their signatures. As many have proclaimed, those days are coming to an end.

A defense department official recently quipped, "If you offer the same protection for your toothbrushes and your diamonds you are bound to lose fewer toothbrushes and more diamonds."  We think that data center security similarly demands specialized solutions.  The concept of an enterprise network has become so ambiguous when it comes to endpoints and devices and supply chain partners, etc. we think its time to think more realistically in terms of trusted, yet highly available zones within the data center.

It seems clear at this point that different parts of the network need very different security capabilities.  Servers, for example need highly accurate solutions that do not block or impede good traffic and can correct bad traffic, especially when it comes to closing network-facing vulnerability windows.  They need to maintain availability with minimal latency for starters; and that has been a sort of Achilles heel for signature-based approaches.  Of course, signatures also bring considerable management burdens over and beyond their security capabilities.

No one is advocating turning off the IPS, but rather approaching servers with more specialized capabilities.  We started focusing on servers years ago and established very sophisticated application and protocol intelligence, which has allowed us to correct traffic inline without the noise, suspense and delay that general purpose network security appliance users have come to expect.

IPS solutions depend on deep packet inspection typically at the perimeter based on regexp pattern matching for exploits.  Emerging challenges with this approach have made alert and block modes absolutely necessary as most IPS solutions aren’t accurate enough to be trusted in full library block. 

Blue Lane uses a vastly different approach.  We call it deep flow inspection/correction for known server vulnerabilities based on stateful decoding up to layer 7.  We can alert, block and correct, but most of are deployments are in correct mode, with our full capabilities enabled. From an operational standpoint we have substantially different impacts.

A typical IPS may have 10K signatures while experts recommend turning on just a few hundred.  That kind of marketing shell game (find out what really works) means that there will be plenty of false alarms, false positives and negatives and plenty of tuning.  With polymorphic attacks signature libraries can increase exponentially while not delivering meaningful improvements in protection. 

Blue Lane supports about 1000 inline security patches across dozens of very specific server vulnerabilities, applications and operating systems.  We generate very few false alarms and minimal latency.  We don’t require ANY tuning.  Our customers run our solution in automated, correct mode.

The traditional static signature IPS category has evolved into an ASIC war between some very capable players for the reasons we just discussed.Exploding variations of exploits and vectors means that exploit-centric approaches will require more processing power.

Virtualization is pulling the data center into an entirely different direction, driven by commodity processors.  So of course our VirtualShield solution was a much cleaner setup with a hypervisor; we can plug into the hypervisor layer and run on top of existing hardware, again with minimal latency and footprint.

You don’t have to be a Metasploit genius to evade IPS signatures.  Our higher layer 7 stateful decoding is much more resilient. 

2) With zero-days on the rise, pay-for-play vulnerability research and now Zero-Bay (WabiSabiLabi) vulnerability auctions and the like, do you see an uptake in customer demand for vulnerability shielding solutions?

Exploit-signature technologies are meaningless in the face of evanescent, polymorphic threats, resulting in 0-day exploits. Slight modifications to signatures can bypass IPSes, even against known vulnerabilities.  Blue Lane technology provides 0-day protection for any variant of an exploit against known vulnerabilities.  No technology can provide ultimate protection against 0-day exploits based on 0-day vulnerabilities. However, this requires a different class of hacker.

3) As large companies start to put their virtualization strategies in play, how do you see customers addressing securing their virtualized infrastructure?  Do they try to adapt existing layered security methodologies and where do these fall down in a virtualized world?

I’ve explored this topic in depth at the Next Generation Data Center conference last week. Also, your readers might be interested in listening to a recent podcast: The Myths and Realities of Virtualization Security: An Interview. 

To summarize, there are a few things that change with virtualization, that folks need to be aware of.  It represents a new architecture.  The hypervisor layer represents the un-tethering and clustering of VMs, and centralized control.  It introduces a new virtual network layer.  There are entirely new states of servers, not anticipated by traditional static security approaches (like instant create, destroy, clone, suspend, snapshot and revert to snapshot). 

Then you’ll see unprecedented levels of mobility and new virtual appliances and black boxing of complex stacks including embedded databases.  Organizations will have to work out who is responsible for securing this very fluid environment.  We’ll also see unprecedented scalability with Infiniband cores attaching LAN/SAN out to 100’s of ESX hypervisors and thousands of VMs.

Organizations will need the capability to shield these complex, fluid environments; because trying to keep track of individual VMs, states, patch levels, locations will make tuning an IPS for polymorphic attacks look like childs play in comparison.   Effective solutions will need to be highly accurate, low latency solutions deployed in correct mode. Gone will be the days of man-to-man blocking and tuning.  Here to stay are the days of zone defense.

4) VMware just purchased Determina and intends to integrate their memory firewall IPS product as an ESX VM plug-in.  Given your early partnership with VMware, are you surprised by this move?  Doesn’t this directly compete with the VirtualSheild offering?

I wouldn’t read too much into this. Determina hit the wall on sales, primarily because it’s original memory wall technology was too intrusive, and fell short of handling new vulnerabilities/exploits.

This necessitated the LiveShield product, which required ongoing updates, destroying the value proposition of not having to touch servers, once installed. So, this is a technology/people acquisition, not a product line/customer-base acquisition.

VMware was smart to get a very bright set of folks, with deep memory/paging/OS, and a core technology that would do well to be integrated into the hypervisor for the purpose of hypervisor hardening, and interVM isolation. I don’t see VMware entering the security content business soon (A/V, vulnerabilities, etc.). I see Blue Lane’s VirtualShield technology integrated into the virtual networking layer (vSwitch), as a perfect complement to anything that will come out of the Determina acquisition.

5) Citrix just acquired XenSource.  Do you have plans to offer VirtualShield for Xen? 

A smart move on Citrix’s part to get back into the game. Temporary market caps don’t matter. Virtualization matters. If Citrix can make this a two or three horse race, it will keep the VMware, Citrix, Microsoft triumvirate on their toes, delivering better products, and net good for the customer.

Regarding BlueLane, and Citrix/Xensource, we will continue to pay attention to what customers are buying as they virtualize their data centers. For now, this is a one horse show 🙂

"Virtualization and the End of Network Security As We Know It…
The feel good hit of the summer!"

Ye olde blog gets pinged quite a lot with searches and search engine redirects for folks looking for basic virtualization and virtualized security information. 

I had to drum up a basic high-level virtualization security presentation for the ISSA Charlotte Metro gathering back in April and I thought I may as well post it.

It’s in .PDF format.  If you want it in .PPT or Keynote, let me know, I’ll be glad to send it to you.  If it’s useful or you need some explanation regarding the visual slides, please get back to me and I’ll be more than glad to address anything you want.  I had 45 minutes to highlight how folks were and might deal with "securing virtualization by virtualizing security."

Yes, some of it is an ad for the company I used to work for who specializes in virtualized security service layers (Crossbeam) but I’m sure you can see how it is relevant in the preso.  You’ll laugh, you’ll cry, you’ll copy/paste the text and declare your own brilliance.  Here’s the summary slide so those of you who haven’t downloaded this yet will know the sheer genius you will be missing if you don’t:

At any rate, it’s not earth shattering but does a decent job at the high level of indicating some of the elements regarding virtualized security. I apologize for the individual animation slide page build-ups.  I’ll re-upload without them when I can get around to it. (Ed: Done.  I also uploaded the correct version 😉

Here’s the PDF.

/Hoff

(As of 11pm EST — 5.5 hours later 1:45pm EST the next day, you lot have downloaded this over 150 380 times.  Since there are no comments, it’s either the biggest piece of crap I’ve ever produced or you are all just so awe stricken you are unable to type.  Newby, you are not allowed to respond to this rhetorical question…)

Hot on the trails of becoming gigagillionaires, the folks at VMware make my day with this.  Congrats to the folks @ Determina.

Methinks that for the virtualization world, it’s a very, very good thing.  A step in the right direction.

I’m going to prognosticate that this means that Citrix will buy Blue Lane or Virtual Iron next (see bottom of the post) since their acquisition of XenSource leaves them with the exact same problem that this acquisition for VMware tries to solve:

VMware Inc., the market leader in virtualization software, has acquired
Determina Inc., a Silicon Valley maker of host intrusion prevention
products.

…the security of virtualized
environments has been something of an unknown quantity due to the
complexity of the technology and the ways in which hypervisors interact
with the host OS.  Determina’s technology is designed specifically to protect the OS
from malicious code, regardless of the origin of the attack, so it
would seem to be a sensible fit for VMware, analysts say.

In his analysis of the deal, Gartner’s MacDonald sounded many of
the same notes. "By potentially integrating Memory Firewall into the
ESX hypervisor, the hypervisor itself can provide an additional level
of protection against intrusions. We also believe the memory protection
will be extended to guest OSs as well: VMware’s extensive use of binary
emulation for virtualization puts the ESX hypervisor in an advantageous
position to exploit this style of protection," he wrote.

I’ve spoken a lot recently  about how much I’ve been dreading the notion that security was doomed to repeat itself with the accelerated take off of server virtualization since we haven’t solved many of the most basic security problem classes.  Malicious code is getting more targeted and more intelligent and when you combine an emerging market using hot technology without an appropriate level of security… 

Basically, my concerns have stemmed from the observation that if we can’t do a decent job protecting physically-seperate yet interconnected network elements with all the security fu we have, what’s going to happen when the "…network is the computer" (or vice versa.)  Just search for "virtualization" via the Lijit Widget above for more posts on this…

Some options for securing virtualized guest OS’s in a VM are pretty straight foward:

1. Continue to deploy layered virtualized security services across VLAN segments of which each VM is a member (via IPS’s, routers, switches, UTM devices…)
2. Deploy software like Virtual Iron’s which looks like a third party vSwitch IPS on each VM
3. Integrate something like Blue Lane’s ESX plugin-in which interacts with and at the VMM level
4. As chipset level security improves, enable it
5. Deploy HIPS as part of every guest OS.

Each of these approaches has its own sets of pros and cons, and quite honestly, we’ll probably see people doing all five at the same time…layered defense-in-depth.  Ugh.

What was really annoying to me, however, is that it really seemed that in many cases, the VM solution providers were again expecting that we’d just be forced to bolt security ON TO our VM environments instead of BAKING IT IN.  This was looking like a sad reality.

I’ll get into details in another post about Determina’s solution, but I am encouraged by VMware’s acquisition of a security company which will be integrated into their underlying solution set.  I don’t think it’s  a panacea, but quite honestly, the roadmap for solving these sorts of problems were blowing in the wind for VMware up until this point.

"Further, by
using the LiveShield capabilities, the ESX hypervisor could be used
‘introspectively’ to shield the hypervisor and guest OSs from attacks
on known vulnerabilities in situations where these have not yet been
patched. Both Determina technologies are fairly OS- and
application-neutral, providing VMware with an easy way to protect ESX
as well as Linux- and Windows-based guest OSs."

Quite honestly, I hoped they would have bought Blue Lane since the ESX Hypervisor is now going to be a crowded space for them…

We’ll see how well this gets integrated, but I smiled when I read this.

Oh, and before anyone gets excited, I’m sure it’s going to be 100% undetectable! 😉

/Hoff

This will be short and sweet.  Citrix’s announcement that they will clip a swell $500 Million to acquire XenSource on the tail of VMware’s IPO makes nothing but sense.  The timing is interesting; waiting for VMware’s IPO both validated the move but one has to wonder if it jacked the price any.

I can’t wait to see how this maps out over time across Citrix’s product lines which are still fairly siloed at this point.  Leveraging XenSource’s technology is a force multiplier across many elements of their offerings. It’s clear what the first moves will be, but I’m really interested in the longer term play.

At any rate, this is a fantastic strategic move for Citrix; these guys are poised to continue their march to take on Cisco as they become a robust platform for application and content delivery.*   If you take a look at their M&A activity over the last few years, it’s on a direct collision course with Cisco in many vectors. 

The big difference is, you can bolt their solution on instead of having to bake it in and these guys already have a footprint and expertise in the server and client consolidation markets.

Orthogonally, I wonder what effect this might have on f5?  Any thoughts there?

Then there’s Microsoft.  This may be a huge opportunity for other players such as SWsoft  to reinforce defensive positioning by shoring up relationships that otherwise might have gone XS’s way.

It’s going to get messy boys and girls.

This acquisition certainly has its challenges, but it really positions Citrix with as a complement to their existing product offerings.

/Hoff

*It gets more interesting strategically from a defensive position given Cisco’s recent investment of $150M in VMware prior to their IPO and my commentary on the matter here.

I’m a big advocate of software as a service (SaaS) — have been for years.  This evangelism started for me almost 5 years ago when I become a Qualys MSSP customer listening to Philippe Courtot espouse the benefits of SaaS for vulnerability management.  This was an opportunity to allow me to more efficiently, effectively and cheaply manage my VA problem.  They demonstrated how they were good custodians of the data (my data) that they housed and how I could expect they would protect it.

I did not, however, feel *more* secure because they housed my VA data.  I felt secure enough that how they housed it should not fall into the wrong hands.  It’s called an assessment of risk and exposure.  I performed it and was satisfied it matched my company’s appetite and business requirements.

Not one to appear unclear on where I stand, I maintain that the SaaS can bring utility, efficiency, cost effectiveness, enhanced capabilities and improved service levels to a corporation depending upon who, what, why, how, where and when the service is deployed.  Sometimes it can bring a higher level of security to an organization, but so can an armed squadron of pissed off armed Oompa Loompa’s — it’s all a matter of perspective.

I suggest that attempting to qualify the benefits of SaaS by generalizing in any sense is, well, generally a risky thing to do.  It often turns what could be a valid point of interest into a point of contention.

Such is the case with a story I read in a UK edition of IT Week by Phil Muncaster titled "On Demand Security Issues Raised."  In this story, the author describes the methods in which the security posture of SaaS vendors may be measured, comparing the value, capabilities and capacity of the various options and the venue for evaluating an SaaS MSSP:  hire an external contractor or rely on the MSSP to furnish you the results of an internally generated assessment.

I think this is actually a very useful and valid discussion to have — whom to trust and why?  In many cases, these vendors house sensitive and sometimes confidential data regarding an enterprise, so security is paramount.  One would suggest that anyone looking to engage an MSSP of any sort, especially one offering a critical SaaS, would perform due diligence in one form or another before signing on the dotted line.

That’s not really what I wanted to discuss, however.

What I *did* want to address was the comment in the article coming from Andy Kellett, an analyst for Burton, that read thusly:

"Security is probably less a problem than in the end-user organisations
because [on-demand app providers] are measured by the service they provide,"
Kellett argued.

I *think* I probably understand what he’s saying here…that security is "less of a problem" for an MSSP because the pressures of the implied penalties associated with violating an SLA are so much more motivating to get security "right" that they can do it far more effectively, efficiently and better than a customer.

This is a selling point, I suppose?  Do you, dear reader, agree?  Does the implication of outsourcing security actually mean that you "feel" or can prove that you’re more secure or better secured than you could do yourself by using a SaaS MSSP?

"I don’t agree the end-user organisation’s pen tester of choice
should be doing the testing. The service provider should do it and make that
information available."

Um, why?  I can understand not wanting hundreds of scans against my service in an unscheduled way, but what do you have to hide?  You want me to *trust* you that you’re more secure or holding up your end of the bargain?  Um, no thanks.  It’s clear that this person has never seen the results of an internally generated PenTest and how real threats can be rationalized away into nothingness…

Clarence So of Salesforce.com
agreed, adding that most chief information officers today understand that
software-as-a-service (SaaS) vendors are able to secure data more effectively
than they can themselves.

Really!?  It’s not just that they gave into budget pressures, agreed to transfer the risk and reduce OpEx and CapEx?  Care to generalize more thoroughly, Clarence?  Can you reference proof points for me here?  My last company used Salesforce.com, but as the person who inherited the relationship, I can tell you that I didn’t feel at all more "secure" because SF was hosting my data.  In fact, I felt more exposed.

"I’m sure training companies have their own motives for advocating the need
for in-house skills such as penetration testing," he argued. "But any
suggestions the SaaS model is less secure than client-server software are well
wide of the mark."

…and any suggestion that they are *more* secure is pure horsecock marketing at its finest.  Prove it.  And please don’t send me your SAS-70 report as your example of security fu.

So just to be clear, I believe in SaaS.  I encourage its use if it makes good business sense.  I don’t, however, agree that you will automagically be *more* secure.  You maybe just *as* secure, but it should be more cost-effective to deploy and manage.  There may very well be cases (I can even think of some) where one could be more or less secure, but I’m not into generalizations.

Whaddya think?

/Hoff

On the tail of my posts from a week or so ago regarding to Cisco’s Data Center 3.0 announcement, Mr. Chamber’s keynote at VMWorld and the follow-on $150Million investment in VMware, here’s something that really gets my goose honking because the force is strong with this one…

Virtualization.info broke the news last week that VMware will "…allow 3rd party vendors to develop their virtual
switches for ESX Server virtual network, and Cisco is expected to be
the first company announcing such product (Virtual Catalyst?)"

This may sound like a no-brain yawner, but it’s quite profound…not just for Cisco, but for any of the switch vendors who want in on the lucrative virtualization market.

For a quick refresher, let’s review the concept of virtual switches (vSwitches).  From VMware’s definition:

A virtual switch, vSwitch,
works much like a physical Ethernet switch. It detects which virtual
machines are logically connected to each of its virtual ports and uses
that information to forward traffic to the correct virtual machines. A
vSwitch can be connected to physical switches using physical Ethernet
adapters, also referred to as uplink adapters, to join virtual networks
with physical networks. This type of connection is similar to
connecting physical switches together to create a larger network. Even
though a vSwitch works much like a physical switch, it does not have
some of the advanced functionality of a physical switch. For more
information on vSwitches, see Virtual Switches.

Given my previous posts on the matter, this offers two interesting and profound perspectives on the virtualization front:

1. If you recall, I blogged back in February about my participation in a Goldman Sachs Security conference where Jayshree Ullal presented Cisco’s vision of virtualized security.  During the Q&A period after her presentation, I asked her a somewhat loaded question that went something like this:

   
   If now we see the consolidation of multiple OS and applications on a
   single VM host in which the bulk of traffic and data interchange is
   between the VM’s themselves and utilize the virtual switching fabrics (ed: software)
   in the VM Host and never hit the actual physical network
   infrastructure, where, exactly, does this leave the self-defending
   "network" without VM-level security functionality at the "micro
   perimeters" of the VM’s?

   I think that this announcement pretty much answers this question.  Cisco will take the concept that I blogged about previously wherein they will abstract the software from the hardware and provide a virtualized version of a catalyst as the ESX vSwitch.  I wager we will see a subset of security functionality in the vSwitch natively that one might expect in the "physical" Catalyst hardware products as much of the capabilities still hinge on new components such as the ACE.

   Now, if the virtual switch is Cisco’s, you can expect a bevy of interaction between the "virtual switch(es)" and the physical ones that the VM Hosts connect to.  This would provide interfaces between all manner of network controls and monitoring capacities such as firewalls, IDS, IPS, SEIM, and solve the issue above by merely "offloading" this functionality via API’s to the physical boxes plumbed into the network.

   Combine that with NAC agents on the hosts and…whether or not it actually works is neither here nor there.  They told they story and here it is.  It’s good to be king.

2. This brings us to point numero dos…and it’s a doozy.  If you think that the current crop of L2/L3 switching and routing infrastructure is fragile enough, just imagine how much fun it’s going to be trying to detect and defend against infrastructure attacks on virtual switches that open up the guts of the VM hosts and hypervisors to third parties.

   We won’t need a Blue Pill, I’ll take one of these below, instead (it’s a cyanide capsule, btw):

   
   Ettercap and arp-twiddling, anyone?  If you don’t have the capability to virtualize the functional equivalent of IDS taps and/or utilize "IPS" plugins to the hypervisors, compromising a single guestOS on a VM could spell disaster that goes undetected.  We already have issues protecting physically isolated critical infrastructure, can you imagine how much fun this is going to be? 

   I’m not talking about application layer attacks here, I’m talking layer 2/layer 3.  The vicious circle begins anew.  You’ll be worrying about XSS and AJAX attacks on your virtualized web servers whilst the same attacks from 10 years ago will give your shiny new virtual infrastructure a wedgie.

   And since it’s likely we’ll see a repeat of architectural car crashes as we have in the past, most of the inter-VM traffic won’t be mutually authenticated or encrypted, either.  So you’ve got that going for you…

So, I think that this model is what Reflex was aiming for with their vIPS (from Virtual Iron) software for the virtual switch which I blogged about here, but Cisco’s going to one-up them because of their investment in VMware, their switching acumen and the unfair advantage of owning both the virtual/logical switching/routing plane as well as the physical.

Good times are comin’, for sure.  I’m trying not to be cynical.  I think it’s fairly obvious as to what ought to be done to secure this mess before it becomes one, but I’m not sure we’re going to be able to step out in front of this train and stop it before it reaches the station.

/Hoff

I’m sorry, what?

EOM.

/Hoff