Rational Survivability

Greg Ness touched off an interesting discussion when he asked “Will Virtualization Undermine Network Equipment Vendors?”  It’s a great read summarizing how virtualization (and Cloud) are really beginning to accelerate how classical networking equipment vendors are re-evaluating their portfolios in order to come to terms with these disruptive innovations.

I’ve written so much about this over the last three years and my response is short and sweet:

Virtualization has actually long been an enabler for network equipment vendors — not server virtualization, mind you, but network virtualization.  The same goes in the security space. The disruption caused by server virtualization is only acting as an accelerant — pushing the limits of scale, redefining organizational and operational boundaries, and acting as a forcing function causing wholesale reconsideration of archetypal network (and security) topologies.

The compressed timeframe associated with the disruption caused by virtualization and its adoption in conjunction with the arrival of Cloud Computing may seem unnatural given the relatively short window associated with its arrival, but when one takes the longer-term view, it’s quite natural.  We’ve seen it before in vignettes across the evolution of computing, but the convergence of economics, culture, technology and consumerism have amplified its relevance.

To answer Greg’s question, Virtualization will only undermine those network equipment vendors who were not prepared for it in the first place.  Those that were building highly virtualized, context-enabled routing, switching and security products will embrace this swing in the hardware/software pendulum and develop hybrid solutions that span the physical and virtual manifestations of what the “network” has become.

As I mentioned in my blog titled “Quick Bit: Virtual & Cloud Networking – Where It ISN’T Going…”

Specifically, as it comes to understanding how the network plays in virtual and Cloud architectures, it’s not where the network *is* in the increasingly complex virtualized, converged and unified computing architectures, it’s where networking *isn’t.*

Where ISN'T The Network?

Take a look at your network equipment vendors.  Where do they play in that stack above?  Compare and contrast that with what is going on with vendors like Citrix/Xen with the Open vSwitch, Vyatta, Arista with vEOS and Cisco with the Nexus 1000v*…interesting times for sure.

/Hoff

*Disclosure: I work for Cisco.

Those crafty Citrix chaps are at it again.

Last month I reported from Citrix Synergy about discussions I had with Simon Crosby and Ian Pratt about the Citrix/Xen Openswitch which is Citrix’s answer to the Cisco Nexus 1000v married to VMware’s vSphere.

Virtualization.com this morning reported that Vyatta — who describe themselves as the “open source alternative to Cisco” — just raised another round of funding, but check out who’s leading it:

Vyatta today announced it has completed its $10 million Series C round of financing led by Citrix Systems. The new funding round also includes existing investors, Comcast Interactive Capital, Panorama Capital, and ArrowPath Venture Partners. As part of the investment, Gordon Payne, senior vice president and general manager of the Delivery Systems Division at Citrix, has joined the Vyatta Board of Directors where he will assist the company in its next phase of development.

Today, Vyatta also announced that it has joined the Citrix Ready product verification program to create solutions for customers deploying cloud computing infrastructures.

Vyatta will use the funds for operating capital as the company scales its sales efforts and accelerates growth across multiple markets.

Vyatta runs on standard x86 hardware and can be virtualized with modern hypervisors, including the Citrix XenServer™ virtualization platform. Vyatta delivers a full set of networking features that allow customers to connect, protect, virtualize, and optimize their networks, improving performance, reducing costs, and increasing manageability and flexibility over proprietary networking solutions. Vyatta has been deployed by hundreds of customers world-wide in both virtual and non-virtual environments.

This is very, very interesting stuff indeed and it’s clear where Citrix has its sights aimed.  This will be good for customers, regardless of platform because it’s going to drive innovation even further.

The virtual networking stacks — and what they enable — are really going to start to drive significant competitive advantage across virtualization and Cloud vendors.  It’s ought to give customers significant pause when it comes to thinking about their choice of platform and integration.

Nicely executed move, Mr. Crosby.

/Hoff

Simon Crosby and I were interviewed by Mike Mimoso of SearchSecurity.com at the RSA conference.  This was after a panel at the America’s Growth Capital conference and prior to our debate which included Steve Herrod of VMware.

It’s a two-part video that got a bit munged when the cameraman let the tape run out about 1/2 way through 😉

Part 1 can be found here.

Part 2 can be found here.

In my Four Horsemen presentation, I made reference to one of the challenges with how the networking function is being integrated into virtual environments.  I’ve gone on to highlight how this is exacerbated in Cloud networking, also.

Specifically, as it comes to understanding how the network plays in virtual and Cloud architectures, it’s not where the network *is* in the increasingly complex virtualized, converged and unified computing architectures, it’s where networking *isn’t.*

What do I mean by this?  Here’s a graphical representation that I built about a year ago.  It’s well out-of-date and overly-simplified, but you get the picture:

There’s networking at almost every substrate level — in the physical and virtual construct.  In our never-ending quest to balance performance, agility, resiliency and security, we’re ending up with a trade-off I call simplexity: the most complex simplicity in networking we’ve ever seen.   I wrote about this in a blog post last year titled “The Network Is the Computer…(Is the Network, Is the Computer…)”

If you take a look at some of the more recent blips to appear on the virtual and Cloud networking  radar, you’ll see examples such as:

• Cisco’s UCS with converged 10Gb/s Ethernet/FCOE including the Nexus 1000v virtual switch and VN-Link (and upcoming VN-Tag)
• The Forthcoming Citrix/Xen/KVM Virtual Networking Stack 
• Blade Network Technology’s SmartConnect™ and VMready™ 
• Arista Networks Cloud Networking Solutions
• Netronome’s Network Flow Processors for Unified Computing 

This list is far from inclusive.  Yes, I know I’ve left off blade server manufacturers and other players like HP (ProCurve) and Juniper, etc.  as well as ADC vendors like f5.  It’s not that I don’t appreciate their solutions, it’s just that I have a couple of free cycles to write this, and the list above appear on the top of my stack.

I plan on writing in more detail about the impact some of these technologies are having on next generation datacenters and Cloud deployments, as it’s a really interesting subject for me coming from my background at Crossbeam.

The most startling differences are in the approach of either putting the networking (and all its attendant capabilities) back in the hands of the network folks or allowing the server/virtual server admins to continue to leverage their foothold in the space and manage the network as a component of the converged and virtualized solution as a whole.

My friend @aneel (Twitter)  summed it up really well this morning when comparing the Blade Network Technology VMready offering and the Cisco Nexus 100ov:

huh.. where cisco uses nx1kv to put net control more in hands of net ppl, bnt uses vmready to put it further in server/virt admin hands

Looking at just the small sampling of solutions above, we see the diversity in integrated networking, external fabrics, converged fabrics (including storage) and add-on network processors.

It’s going to be a wild ride kids.  Buckle up.

/Hoff

I was at Citrix Synergy/Virtualization Congress earlier this week and at the end of the day on Wednesday, Scott Lowe tweeted something interesting when he said:

In my mind, the biggest announcement that no one is talking about is virtual switching for XenServer. #CitrixSynergy

I had missed the announcements since I didn’t get to many of the sessions due to timing, so I sniffed around based on Scott’s hints and looked for some more meat.

I found that Chris Wolf covered the announcement nicely in his blog here but I wanted a little more detail, especially regarding approach, architecture and implementation.

Imagine my surprise when Alessandro Perilli and I sat down for a quick drink only to be joined by Simon Crosby and Ian Pratt.  Sometimes, membership has its privileges 😉

I asked Simon/Ian about the new virtual switch because I was very intrigued, and since I had direct access to the open source, it was good timing.

Now, not to be a spoil-sport, but there are details under FreiNDA that I cannot disclose, so I’ll instead riff off of Chris’ commentary wherein he outlined the need for more integrated and robust virtual networking capabilities within or adjunct to the virtualization platforms:

Cisco had to know that it was only a matter of time before competition for the Nexus 1000V started to emerge, and it appears that a virtual switch that competes with the Nexus 1000V will come right on the heels of the 1000V release. There’s no question that we’ve needed better virtual infrastructure switch management, and an overwhelming number of Burton Group clients are very interested in this technology. Client interest has generally been driven by two factors:

• Fully managed virtual switches would allow the organization’s networking group to regain control of the network infrastructure. Most network administrators have never been thrilled with having server administrators manage virtual switches.
• Managed virtual switches provide more granular insight into virtual network traffic and better integration with the organization’s existing network and security management tools

I don’t disagree with any of what Chris said, except that I do think that the word ‘compete’ is an interesting turn of phrase.

Just as the Cisco 1000v is a mostly proprietary (implementation of a) solution bound to VMware’s platform, the new Citrix/Xen/KVM virtual networking capabilities — while open sourced and free — are bound to Xen and KVM-based virtualization platforms, so it’s not really “competitive” because it’s not going to run in VMware environments. It is certainly a clear shot across the bow of VMware to address the 1000v, but there’s a tradeoff here as it comes to integration and functionality as well as the approach to what “networking” means in a virtualized construct.  More on that in a minute.

I’m going to take Chris’ next chunk out of order in order to describe the features we know about:

I’m expecting Citrix to offer more details of the open source Xen virtual switch in the near future, but in the mean time, here’s what I can tell you:

• The virtual switch will be open source and initially compatible with both Xen- and KVM-based hypervisors
• It will provide centralized network management
• It will support advanced network management features such as Netflow, SPAN, RSPAN, and ERSPAN
• It will initially be available as a plug-in to XenCenter
• It will support security features such as ACLs and 802.1x

This all sounds like good stuff.  It brings the capabilities of virtual networking and how it’s managed to “proper” levels.  If you’re wondering how this is going to happen, you *cough* might want to take a look at OpenFlow…being able to enforce policies and do things similar to the 1000v with VMware’s vSphere, DVS and the up-coming VN-Link/VN-tag is the stuff I can’t talk about — even though it’s the most interesting.  Suffice it to say there are some very interesting opportunities here that do not require proprietary networking protocols that may or may not require uplifts or upgrades of routers/switches upstream.  ’nuff said. 😉

Now the next section is interesting, but in my opinion is a bit of reach in certain sections:

For awhile I’ve held the belief that the traditional network access layer was going to move to the virtual infrastructure. A large number of physical network and security appliance vendors believe that too, and are building or currently offering products that can be deployed directly to the virtual infrastructure. So for Cisco, the Nexus 1000V was important because it a) gave its clients functionality they desperately craved, but also b) protected existing revenue streams associated with network access layer devices. Throw in an open source managed virtual switch, and it could be problematic for Cisco’s continued dominance of the network market. Sure, Cisco’s competitors can’t go at Cisco individually, but by collectively rallying around an open source managed virtual switch, they have a chance. In my opinion, it won’t be long before the Xen virtual switch can be run via software on the hypervisor and will run on firmware on SR-IOV-enabled network interfaces or converged network adapters (CNAs).

…

This is clearly a great move by Citrix. An open source virtual switch will allow a number of hardware OEMs to ship a robust virtual switch on their products, while also giving them the opportunity to add value to both their hardware devices (e.g., network adapters) and software management suites. Furthermore, an open source virtual switch that is shared by a large vendor community will enable organizations to deploy this virtual switch technology while avoiding vendor lock-in.

Firstly, I totally agree that it’s fantastic that this capability is coming to Xen/KVM platforms.  It’s a roadmap item that has been missing and was, quite honestly, going to happen one way or another.

You can expect that Microsoft will also needto respond to this some point to allow for more integrated networking and security capabilities with Hyper-V.

However, let’s compare apples to apples here.

I think it’s interesting that Chris chose to toss in the “vendor lock-in” argument as it pertains to virtual networking and virtualization for the following reasons:

• Most enterprise networking environments (from the routing & switching perspective) are usually provided by a single vendor.
• Most enterprises choose a virtualization platform from a single vendor

If you take those two things, then for an environment that has VMware and Cisco, that “lock-in” is a deliberate choice, not foisted upon them.

If an enterprise chooses to invest based upon functionality NOT available elsewhere due to a tight partnership between technology companies, it’s sort of goofy to suggest lock-in.  We call this adoption of innovation.  When you’re a competitor who is threatened because don’t have the capability you call it lock-in. ;(

This virtual switch announcement does nothing to address “lock-in” for customers who choose to run VMware with a virtual networking stack other than VMware’s or Cisco’s…see what I mean.  it doesn’t matter if the customer has Juniper switches or not in this case…until you can integrate an open source virtual switch into VMware the same way Cisco did with the 1000v (which is not trivial,) then we are where we are.

Of course the 1000v was a strategic decision by Cisco to help re-claim the access layer that was disappering into the virtualized hosts and make Cisco more relevant in a virtualized environment.  It sets the stage, as I have mentioned, for the longer term advancements of the entire Nexus and NG datacenter switching/routing products including the VN-Link/VN-Tag — with some features being proprietary and requiring Cisco hardware and others not.

I just don’t buy the argument that an open virtual switch “… could be problematic for Cisco’s continued dominance of the network market.” when the longtime availablity of open source networking products (including routers like Vyatta) haven’t made much of a dent in the enterprise against Cisco.

Customers want “open enough” and solutions that are proven and time tested.  Even the 1000v is brand new.  We haven’t even finished letting the paint dry there yet!

Now, I will say that if IBM or HP want to stick their thumb in the pie and extend their networking reach into the host by integrating this new technology with their hardware network choices, it offers a good solution — so long as you don’t mind *cough* “lock-in” from the virtualization platform provider’s perspective (since VMware is clearly excluded — see how this is a silly argument?)

The final point about “security inspection” and comparing the ability to redirect flows at a kernel/network layer to a security VA/VM/appliance  is only one small part of what VMware’s VMsafe does:

Citrix needed an answer to the Nexus 1000V and the advanced security inspection offered by VMsafe, and there’s no doubt they are on the right track with this announcement.

Certainly, it’s the first step toward better visibility and does not require API modification of the security virtual appliances/machines like VMware’s solution in it’s full-blown implementation does, but this isn’t full-blown VM introspection, either.

Moreso, it’s a way of ensuring a more direct method of gaining better visibility and control over networking in a virtualized environment.  Remember that VMsafe also includes the ability to provide interception and inspection of virtualized memory, disk, CPU execution as well as networking.  There are, as I have mentioned Xen community projects to introduce VM introspection, however.

So yes, they’re on the right track indeed and will give people pause when evaluating which virtualization and network vendor to invest in should there be a greenfield capability to do so.  If we’re dealing with environments that already have Cisco and VMware in place, not so much.

/Hoff

I’ll be at the Virtualization Congress ’09 / Citrix Synergy at the MGM Grand in Las Vegas for a couple of days this week.

I am presenting on Cloud Computing Security on May 6th at 11:30am-12:20pm – Mozart’s The Marriage of Figaro: The Complexity and Insecurity of the Cloud – VC105

This ought to be a funny presentation for about the first 5 minutes…you’ll see why 😉

I’m also on a panel with Dave Shackleford (Configuresoft) & Michael Berman (Catbird) moderated by the mastermind of all things virtualization, Alessandro Perelli,  on May 6th at 5: Securing the Virtual Data Center (on Earth and on Clouds) – VC302

If you’re around, ping me via DM on Twitter (@beaker) or hit me up via email [choff @ packetfilter.com]

Of course, it’s entirely likely you’ll find Crosby and I chatting it up somewhere 😉

See you there!

/Hoff

Besides the sumo suit wrestling match I'm organizing between myself and Simon Crosby at this year's coming RSA 2009 show, I'm really excited to announce that there will be another exciting virtualization security (VirtSec) event happening at the show.

"In this verbal cage match session, two well known critics of virtualization security take on two virtualization company CTOs as they spar over how best to secure virtualization platforms: who should be responsible for securing it, and how that ultimately impacts customers and attackers.  We have Hoff and Skoudis versus Crosby and Herrod.  Refereeing will be respected analyst, Antonopoulos."

Simon Crosby and I have been going 'round a bit lately arguing the premise of where, why, when, how and how much security should be invested by either embedding it in the virtualization platform itself or being addressed by third parties.

Simon's last sentence in his latest riposte titled "Hoff is Still Confused" was interesting:

Well, how the hell am I supposed to argue with that!? ;)  OK, now I am confused! Simon's taken the high road and thus I shall try to do so, too.  I wrote a ton more in response, but I'm not sure anybody cares. 😉

All told, I think we're both aiming at a similar goal in spite of our disparate approaches: achieving a more secure virtualized environment.

But seriously, I don't think that I'm confused about Citrix's position on this matter, I just fundamentally disagree with it.

I feel strongly that Simon and I really are on different sides of a religious issue but without a more reasonable platform for discussion, I'm not sure how we'll intelligently discuss this more coherently without all the back and forth.  Perhaps a cage match in sumo suits!?

I appreciate Simon clarifying his position and reaching out to ensure we are on the same page.  We're not, but the book's not closed yet. 

So we agree to disagree, and I respect Simon for his willingness to debate the issue.

/Hoff

Hat-tip to David Marshall for the pointer.

In what can only be described as the natural evolution of Xen's security architecture, news comes of a Xen community project to integrate a VM Introspection API and accompanying security functionality into Xen.  Information is quite sparse, but I hope to get more information from the project leader, Stephen Spector, shortly. (*Update: Comments from Stephen below)

This draws naturally obvious parallels to VMware's VMsafe/vNetwork API's which will yield significant differentiation and ease in integrating security capabilities with VMware infrastructure when solutions turn up starting in Q1'09.

From the Xen Introspection Project wiki:

The
purpose of the Xen Introspection Project is to design an API for
performing VM introspection and implement the necessary functionality
into Xen. It is anticipated that the project will include the following
activities (in loose order): (1) identification of specific
services/functions that introspection should support, (2) discussion of
how that functionality could be achieved under the Xen architecture,
(3) prioritization of functionality and activities, (4) API definition,
and (5) implementation.

It is important to note that this is not the first VMI project for Xen.  There is also the Georgia Tech XenAccess project lead by Bryan Payne which is a library which allows a privileged domain to gain access to the runtime state of another domain.  XenAccess focuses (initially) on memory introspection but is adaptable to disk I/O also:

I wonder if we'll see XenAccess fold into the VMI Xen project?

Astute readers will also remember my post titled "The Ghost of Future's Past: VirtSec Innovation Circa 2002" in which I reviewed work done by Mendel Rosenblum and Tal Garfinkel (both of VMware fame) on the LiveWire project which outlined VMI for isolation and intrusion detection:

What's old is new again.

Given my position advocating VMI and the need for inclusion of this capacity in all virtualization platforms versus that of Simon Crosby, Citrix's (XenSource) CTO in our debate on the matter, I'll be interested to see how this project develops and if Citrix contributes. 

Microsoft desperately needs a similar capability in Hyper-V if they are to be successful in ensuring security beyond VMM integrity in their platform and if I were a betting man, despite their proclivity for open-closedness, I'd say we'll see something to this effect soon.

I look forward to more information and charting the successful evolution of both the Xen Introspection Project and XenAccess.

/Hoff

Update: I reached out to Stephen Spector and he was kind enough to respond to a couple of points raised in this blog (paraphrased from a larger email):

Bryan Payne from Georgia tech will be participating in the project and there is some other work going on at the University of Alaska at Fairbanks. The leader for the project is Stephen Brueckner from NYC-AT.

As for participation, Citrix has people already committed and I have 14 people who have asked to take part.

Sounds like the project is off to a good start! 

I was having an interesting discussion the other evening at BeanSec with Jeanna Matthews from Clarkson University.  Jeanna is one of the authors of what I think is the best book available on Xen virtualization, Running Xen.

In between rounds of libations, the topic of Hypervisor-neutral, VM portability/interoperability between the virtualization players (see right) came up.  If I remember correctly, we were discussing the announcement from Citrix regarding Project Kensho:

Santa Clara, CA » 7/15/2008 » Citrix Systems, Inc.
(Nasdaq:CTXS), the global leader in application delivery
infrastructure, today announced “Project Kensho,” which will deliver
Open Virtual Machine Format (OVF) tools that, for the first time, allow
independent software vendors (ISVs) and enterprise IT managers to
easily create hypervisor-independent, portable enterprise application
workloads. 
These tools will allow application workloads to be imported
and run across Citrix XenServer™, Microsoft Windows Server 2008 Hyper-V™ and VMware™ ESX virtual environments. 

On the surface, this sounded like a really interesting and exciting development regarding interoperability between virtualization platforms and the VMs that run on them.  Digging deeper, however, it’s not really about virtualization at all; it’s about the delivery of applications and services — almost in spite of the virtualization layer — which is something I hinted about at the end of this post.

I am of the opinion that virtualization is simply
a means to an end, a rationalized and cost-driven stepping-stone along the path of
designing, provisioning, orchestrating, deploying, and governing a more agile, real time
infrastructure to ensure secure, resilient, cost-effective and dynamic delivery of service.

You might call the evolution of virtualization and what it’s becoming cloud computing.  You might call it utility computing.  You might call it XaaS.  What many call it today is confusing, complex, proprietary and a pain in the ass to manage.

Thus, per the press release regarding Project Kensho, the notion of packaging applications/operating environments up as tasty little hypervisor-neutral nuggets in the form of standardized
virtual appliances that can run anywhere on any platform is absolutely appealing and in the long term, quite necessary.*

However, in the short term, I am left wondering if this is a problem being "solved" for ISV’s and virtualization platform providers or for customers?  Is there a business need today for this sort of solution and is the technology available to enable it?

Given the fact that my day job and paycheck currently depends upon crafting security strategies, architecture and solutions for real time infrastructure, I’m certainly motivated to discuss this.  Mortgage payment notwithstanding, here’s a doozy of a setup:

Given where we are today with the heterogeneous complexity and nightmarish management realities of our virtualized and non-virtualized infrastructure, does this really solve relevant customer problems today or simply provide maneuvering space for virtualization platform providers who see their differentiation via the hypervisor evaporating?

While the OVF framework was initially supported by a menagerie of top-shelf players in the virtualization space, it should come as no surprise that this really represents the first round in a cage match fight to the death for who wins the application/service delivery management battle.

You can see this so clearly in the acquisition strategies of VMware, Citrix and Microsoft.

Check out the remainder of the press release.  The first half had a happy threesome of Citrix, Microsoft and VMware taking a long walk on the beach.  The second half seems to suggest that someone isn’t coming upstairs for a nightcap:

Added Value for Microsoft Hyper-V

Project Kensho will also enable customers to leverage the
interoperability benefits and compatibility between long-time partners
Citrix and Microsoft to extend the Microsoft platform.  For example,
XenServer is enhanced with CIM-based management APIs to allow any
DMTF-compliant management tool to manage XenServer, including Microsoft
System Center Virtual Machine Manager. And because the tools are based
on a standards framework, customers are ensured a rich ecosystem of
options for virtualization.  In addition, because of the open-standard
format and special licensing features in OVF, customers can seamlessly
move their current virtualized workloads to either XenServer or
Hyper-V, enabling them to distribute virtual workloads to the platform
of choice while simultaneously ensuring compliance with the underlying
licensing requirements for each virtual appliance.

Project Kensho will support the vision of the Citrix Delivery Center™
product family, helping customers transform static datacenters into
dynamic “delivery centers” for the best performance, security, cost
savings and business agility. The tools developed through Project
Kensho will be easily integrated into Citrix Workflow Studio™ based
orchestrations, for example, to provide an automated, environment for
managing the import and export of applications from any major
virtualization platform.

Did you catch the subtlety there?  (Can you smell the sarcasm?)

I’ve got some really interesting examples of how this is currently shaking out in very large enterprises.  I intend to share them with you, but first I have a question:

What relevance do hypervisor-neutral virtual appliance/machine deployments have in your three year virtualization roadmaps?  Are they a must-have or nice-to-have? Do you see deploying multiple hypervisors and needing to run these virtual appliances across any and all platforms regardless of VMM?

Of course it’s a loaded question.  Would you expect anything else?

/Hoff

* There are some really interesting trade-offs to be made when deploying virtual appliances.  This is the topic of my talk at Blackhat this year titled "The Four Horsemen of the Virtualization Apocalypse"