Rational Survivability

Arista Networks is a company stocked with executives whose pedigrees read like the who's-who from the networking metaverse.  The CEO of Arista is none other than Jayshree Ullal, the former senior Vice President at Cisco responsible for their Data Center, Switching and Services and Andres von Bechtolsheim from Sun/Granite/Cisco serves as Chief Development Officer and Chairman.

I set about to understand what business Arista was in and what problems they aim to solve given their catchy (kitchy?) tagline of Cloud Networking™

Arista makes 10GE switches utilizing a Linux-based OS they call EOS which provides high-performance networking. 

The EOS features a "…multi-process state sharing architecture that completely
separates networking state from the processing itself. This enables
fault recovery and incremental software updates on a fine-grain process
basis without affecting the state of the system."

I read through the definition/criteria that describes Arista's Cloud Networking value proposition: scalability, low latency, guaranteed delivery, extensible management and self-healing resiliency.

These seem like a reasonable set of assertions but I don't see much of a difference between these requirements and the transformative requirements of internal enterprise networks today, especially with the adoption of virtualization and real time infrastructure. 

Pawing through their Cloud Networking Q&A, I was struck by the fact that the fundamental assumptions being made by Arista around the definition of Cloud Computing are very myopic and really seem to echo the immaturity of the definition of the "cloud" TODAY based upon the industry bellweathers being offered up as examples of leaders in the "cloud" space.

Let's take a look at a couple of points that make me scratch my head:

That's  a very narrow definition of cloud computing and seems to be rooted in examples of large, centrally-hosted providers today such as those quoted.  This definition seems to be at odds with other cloud computing providers such as 3tera and others who rely on distributed computing resources that may or may not be centrally located.

While I don't disagree that consolidation through server virtualization is not the same thing as cloud computing, the statement that "basically all cloud computing applications today run directly
on a physical server without the use of virtualization or Hypervisors" is simply untrue.

I don't see how that assertion has been formulated or substantiated.

I'm puzzled when I look at Arista's assertion that existing and emerging networking solutions from the likes of Cisco are not capable of providing these capabilities while they simultaneously seem to shrug off the convergence of storage and networking.  Perhaps they simply plan on supporting FCoE over 10GE to deal with this?

Further,  ignoring the (initial) tighter coupling of networkng with virtualization to become more virtualization-aware with the likes of what we see from the Cisco/VMware partnership delivering VN-Link and the Nexus 1000v, Ieaves me shaking my head in bewilderment.

Further, with the oft-cited example of Amazon's cloud model as a reference case for Arista, they seem to ignore the fact that EC2 is based upon Xen and is now offering both virtualized Linux and Windows VM support for their app. stack.

It's unclear to me what problem they solve that distinguishes them from entrenched competitors/market leaders in the networking space unless the entire value proposition is really hinged on lower cost.  Further, I couldn't find much information on who funded (besides the angel round from von Bechtolsheim) Arista and I can't help but wonder if this is another Cisco "spin-in" that is actually underwritten by the Jolly Green Networking Giant.

If you've got any useful G2 on Arista (or you're from Arista and want to chat,) please do drop me a line…

/Hoff

I was reading an interesting article from James Maguire from Datamation that outlined various competitors in the virtualization platform space.

In the article, James referenced a Gartner slide that comparatively summarized hypervisor selection criteria including the maturity of features, pricing, management and ultimately security.  Unfortunately the presentation source of the slide was not cited, but check this out:


What I found very interesting was the security section which equated the security capability/maturity criteria of Oracle with that of VMware while at the same time demonstrating that the overall maturity/stability of Oracle was not has highly ranked. 

Since Oracle's hypervisor is based upon Xen and Citrix/XenSource is not ranked as high, it leaves me scratching my head.

Given that this chart references hypervisor selection to YE08, it more than likely does not take into consideration the coming vNetwork/VMsafe API's; it's unclear if this section is a measure of VMM "security" based upon published vulnerabilities, an assessment of overall architecture, the availability of security solutions in the ecosystem…

This is a very interesting assertion and I'd really like to get the entire document that describes how this was quantified and what it means.  Anyone know which report this came from?

/Hoff

I'm literally emulating a bobble head doll at this point.  In a fit of snarky confusion,  I'm simultaneously trying to nod-shake-shrug my oversize gourd to arrive at some commonsensical conclusion about this piece.  I can't, so my head just flops about like the headpiece on a 4-axis CNC machine.

Tarry Singh from the Avastu Blog spends his time as an independent analyst covering virtualization and cloud computing. His latest post regarding security left me scratching my head.

I had a bunch of folks ping me asking me for my interpretation of Tarry's latest work but I thought I'd turn it over to you lot since the more eyeballs the merrier.

Tarry's post is titled "Good News! Hackers Focus On Virtualization."

I read it.  I read it again.  I had something to drink.  I read half of it.

I think what Tarry's trying to say is that with more attention being paid to virtualization platforms by "hackers" that we ought to see increased pressure for more secure environments due to impending carnage from mounting exploits and regulators amassing mad virtualization audit skillz.  I could be wrong as it was really, really good wine.

Despite abusing the term "hackers," it's not an unreasonable assertion despite being dusty.  The rest of the post (or the wine) still leaves me a bit dizzy.

Pay attention now, I'll highlight the interesting bits in bold…

• This is a validation of the fact that Virtualization is going mainstream
• Security and Compliance will be core focus of all organizations
• Virtual Infrastructures are easier to battendown and secure due to its uniformity
  
• Regulators
  will increasingly ask for audits, where as in traditional environments
  (I've seen such audits by the like of KPMG etc) and always wondered
  like "wow–so are so prepared, dude, NOT!", Virtual environments
  suddenly enables auditors to ask the right questions and get or not get the expected results.
• Focus on security would mean that we will have to work harder to provide a secure and compliant platforms.

I'd be very interested to understand what a "secure and compliant practice" within the scope of a virtualized environment means, especially in light of some of the statements above. 

Tarry, you've got mail.

/Hoff

I was reading Bruce's recent post on Quantum Crypto and couldn't believe what I read.  I'm horrified:

No commercial value? Doesn't solve any security problem that needs solving?  Isn't worth paying for?  Only a few folks buying and deploying it!?

Hell, I'm writing a business plan right now and going for VC funding!  This is obviously the next big thing!  After all, this is mantra that the entire security industry is predicated upon.

Silly Bruce.

/Hoff

One of the things I'm very much looking forward to with the release of Cisco Nexus 1000v virtual switch for ESX is the release of performance figures for the solution.

In my Four Horsemen presentation I highlight with interest the fact that in the physical world today we rely on dedicated, highly-optimized multi-core COTS or ASIC/FPGA-powered appliances to deliver consistent security performance in the multi-Gb/s range. 

These appliances generally deliver a single function (such as firewall, IPS, etc.) at line rate and are relatively easy to benchmark in terms of discrete performance or even when in-line with one another.

When you take the approach of virtualizing and consolidating complex networking and security functions such as virtual switches and virtual (security) appliances on the same host competing for the same compute, memory and scheduling resources as the virtual machines you're trying to protect, it becomes much more difficult to forecast and preduct performance…assuming you can actually get the traffic directed through these virtual bumps in the proper (stateful) order.

Recapping Horsemen #2 (Pestilence,) VMware's recently published performance results (grain of NaCl taken) for ESX 3.5 between two linux virtual machines homed to the same virtual switch/VLAN/portgroup in a host shows throughput peaks of up to 2.5 Gb/s.  Certainly the performance at small packet rates are significantly less but let's pick the 64KB-64KB sampled result shown below for a use case:

Given the performance we see above (internal-to-internal) it will be interesting to see how the retooling/extension of the networking functions to accomodate 3rd party vSwitches, DVS, API's, etc. will affect performance and what overhead these functions impose on the overall system.  Specifically, it will be very interesting to see how VMware's vSwitch performance compares to Cisco's Nexus 1000v vSwitch in terms of "apples to apples" performance such as the test above.*

It will be even more interesting to see what happens when vNetwork API's (VMsafe) API calls are made in conjunction with vSwitch interaction, especially since the performance processing will include the tax of any third party fast path drivers and accompanying filters.  I wonder if specific benchmarking test standards will be designed for such comparison?

Remember, both VMware's and Cisco's switching "modules" are software — even if they're running in the vKernel, so capacity, scale and performance are a function of arbitrated access to hardware via the hypervisor and any hardware-assist present in the underlying CPU.

What about it, Omar?  You have any preliminary figures (comparable to those above) that you can share with us on the 1000v that give us a hint as to performance?

/Hoff

* Further, if we measure performance that benchmarks traffic including
physical NICs, it will be interesting to see what happens when we load
a machine up with multiple 10Gb/s Ethernet NICs at production loads
trafficked by the vSwitches.

In the immortal words of David Byrne:

Look, I love my brother from a different mother, and as entertaining as I find Amrit's latest blog on the end of the world due to the world economic malaise, I can't help but remember the last time this happened at the end of the dot-com bubble. 

You might say that it's never been this bad.  You might be right.  However, we've all weathered storms before and while things certainly change — and not always for the best — security will survive.  It may look a little different, however.  Meh.

As I have both said and experienced previously, situations such as this will deliver new regulations and oversight, more compliance requirements, stretched/reduced budgets and a streamlining in role, process, function and technology.  It's the flatlining function in the pulse before the CPR kicks in.

Amrit's predictions are interesting, but all of these things were happening well BEFORE the financial crisis hit as part of the normal cycle of punctuated equilibrium.  Seriously, we've seen this behavior for the last four years already.*  To paraphrase Amrit's "predictions:"

• Innovation will come to a grinding halt
• Coming regulations will add to compliance madness
• Enterprises will instantiate process/capability maturity and efficiency models
• Companies will move more functions/services to outsourced partners and grapple with SLA, ownership and portability issues.
• Vendors will quickly grasp at the latest buzzword in order to maintain relevance such as virtualization, SaaS, Cloud, etc.

So again, which of these weren't already happening?

Times are tough.  So are we. 

See you Monday.

/Hoff

P.S.  Buried in the comments is the most profound point I have to make in response to Amrit:

You know how I know this isn't the end of the [security] world? You [Amrit] and I — people who make a career by squawking on blogs — still have jobs

* To make it clear, because I've obviously done a poor job understanding Amrit's points, I'm not suggesting that the impacts of the last few months aren't taking a toll.  I'm suggesting, however, that the crisis(es) are acting as an accelerant delivering more quickly the outcomes of things already in motion.  Further, as I mentioned in the comments, while innovation is certainly delivered from the tech. startup community, it's also driven from corporations when necessity pushes for innovation and innovative solutions even due to reasons like cost control…

How’d ya like this picture of “THE Cloud…”

This love affair with abusing the amorphous thing called “THE Cloud” is rapidly  approaching meteoric levels of asininity.  In an absolute fit of angst I make the following comments:

1. There is no singularity that can be described as “THE Cloud.” There are many clouds, they’re not federated, they don’t natively interoperate at the application layer and they’re all mostly proprietary in their platform and operation.  They’re also not all “public” and most don’t exchange data in any form. The notion that we’re all running out to put ALL our content and apps in some common repository on someone else’s infrastructure (or will) is bullshit.  Can we stop selling this lemon already? There will be lots of Clouds that we’ll spread much of our information and applications onto — some internal, some external, some public, some private….

   Yay!  More people have realized that outsourcing operations and reducing both OpEx and CapEx by using shared infrastructure makes sense.  They also seem to have just discovered it has some real thorny issues, too.  Welcome to the 90’s. Bully!Just like there are many types of real billowing humid masses (cumulonimbus, fibratus, undulatus, etc.) there are many instantiations of resource-based computing models that float about in use today — mobile.me, SalesForce.com, Clean Pipes from ISP’s, Google/Google Apps, Amazon EC2, WebEx — all “cloud” services.  The only thing they have in common is they speak a dialect called IP…

2. The current fad of butchering the term “Cloud Computing” to bring sexy back to the *aaS (anything as a service) model is embarrassing. More embarrassing is the fact that I agree with Larry Ellison wherein he stated:

   “The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. I can’t think of anything that isn’t cloud computing with all of these announcements.
   The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s
   insane. When is this idiocy going to stop?“A-Freaking-Men.

3. It ain’t all new, folks. Suggesting that this is a never-before-seen paradigm that we’ve not faced prior and requires entirely thinking as to privacy, trust models, security as a service layer and service levels mocks the fact that the *aaS model is something we’ve been grappling with for years and haven’t answered.  See #2.  I mean really.  I’ve personally been directly involved with cloud-models since the early 90’s.  Besides the fact that it’s become (again) an economically attractive and technologically viable option doesn’t make it new, it makes it convenient and marketable.  That said, we’re going to struggle with the operational and organizational issues and where theory meets practice on the battlefield.
4. Infrastructure Gorillas are clouding the issue by suggesting thier technology represents THE virtual datacenter OS. Microsoft, Citrix, VMware, Cisco.  They all say the same thing using different words.  Each of them claiming ownership as the platform/OS upon which “THE cloud” will operate.  Not one of them have a consistent model of securing their own vDCOS, so don’t start on how we’re going to secure “IT.”(Ed: In fairness just so nobody feels left out, I should also add that the IaaS (Infrastructure as a service)/integrator gorillas such as IBM and HP are also in the mix — each with their own flavor of service differentiation sprinkled on top.)

If you thought virtualization and its attendant buzzwords, issues and spin were egregious, this billowy mass of marketing hysteria is enough to make me…blog 😉

C’mon, people. Don’t give into the generalist hype.  Cloud computing is real.  “THE Cloud?”  Not so much.

/Hoff

(I don’t know what it was about this article that just set this little rant off, but well done Mr. Moyle)

From Virtualization.com comes the news that VMware has acquired BlueLane Technologies

BlueLane is the maker of solutions that protect both physical and logical infrastructure which includes ServerShield and VirtualShield.  The company has of late focused wisely on
the latter which provides application-aware firewalling, inter-VM flow visibility and analytics, application policy control, and intrusion prevention capabilities.

Coupled with the introspection capabilities provided by VMware's vNetwork/VMsafe API's natively, the integration of BlueLane's solution sets will add to the basal capabilities of the platform itself and will allow customers the flexibility to construct more secure virtualized operating environments.

The notion of enabling in-line patch-proxying as well as the "IPS-like" in-line vulnerability mitigation capabilities for VM's and additional VMM protection make this very interesting indeed.  You can read more about BlueLane's approach on their website.  I also interviewed Allwyn Sequeira on my blog.

VMware's acquisition of Blue Lane comes as no surprise as it became clear to me that in order to continue to strengthen the underlying platform of the hypervisor itself, I wrote earlier this month prior to rumors of Blue Lane's acquisition by other bloggers that as part of a successful differentiation strategy:

Of course, I actually talked about it a year ago when Determina was acquired…

I think it's actually an excellent move as it continues on the path of not only helping to ensure that the underlying virtualization platform is more secure, but the elements that ride atop on it are equally "security enabled" also. 

This point was at the heart of my debate with Simon Crosby, Citrix Systems' CTO (see here and here);
focusing solely on VMM resilience and leaving the ISVs to sort out security was a bad idea.  It  leads to more siloes, less integration, more complexity and overall a less secure environment.

We need a unified secure ecosystem to start with instead of worrying about securing the ecosystem's products.

Form a business perspective it takes a mixture of resolve, market dominance, and confidence to cannibalize a section of your ecosystem, but it's the right thing to do in this case in order to offset competitive forces and help customers solve some really nasty issues.

I made mention of this point with emerging security ISV's at Vmworld, and was asked several times whether I really thought VMware would do this.  The odd question that inevitably came next was "were does that leave security ISV's like us?"  You can guess my answer.  Honestly, I'm sure most of them were hoping to be bought for the same reason.

So, will this cause a run on alignment to support Hyper-V over VMware?  I don't think so.  ISV's who were hinging their hopes for success solely on VMware understand this risk.  Microsoft has no API facility like vNetwork/VMsafe, so the options for reasonable and rational installation of their products are limited.  Citrix is in the same boat.

This is the reason my next set of VirtSec presentations will focus on Hyper-V.

On a side note, I was one of Blue Lane's first customers for their patch proxy product and have been an ardent supporter of their approach for many years, despite taking quite a bit of crap for it from purists and pundits who had difficulty rectifying the approach in comparison to traditional IPS'.

This is a good thing for VMware, VMware's customers and Blue Lane. Congratulations to the BlueLane team.

It's been a whirlwind tour recently travel-wise as I've been speaking quite a bit on the virtualization circuit regarding security (or lack thereof.)

I've spent some serious time talking to users, vendors and analysts regarding some of the research I've been doing on current and future state virtualization technologies and roadmaps.

To cap of this year's events, I'll be at SecTor in Toronto on 10/8-9 and DayCon in Dayton from 10/10-12.

After that in November (at Information Security Decisions) I'll be officially retiring the Four Horsemen presentation in lieu of the next in the series to come.

Hope to see you in Toronto or Dayton, eh?

/Hoff

I spend a good amount of time thinking about how multiple technology strategies from market leaders coalesce into a reasonably homogenized version of reality in the networking and security space in order to decide where to place bets; it's akin to reading a tell and analyzing a player's betting strategy at a poker table.

I look at Cisco and VMware and can't help to chuckle at the moves being made in the name of "partnership" on the virtualization front and there's an awful lot of twitching going on that doesn't require Phil Hellmuth to decode. 

Partnerships are nothing new, but usually they are couched with certain modicum of suspicion and cynicism.  However, speaking with folks at VMworld, either folks were high from the Oxygen Bar at the airport, or they were adding naiveté syrup to the drinks because I seemed alone in my concerns…

I've put together a couple of summary points on the matter — more for my own personal enjoyment and note taking than anything else — and framed them in terms of what I find to be really annoyingly obvious examples of these two strange bedfellows' behaviors:

1)  The purported cohesion of Cisco's and VMware's virtualization strategies is a simply a matter of converged parallelism and forced perspective.

You've seen diagrams that demonstrate the notion of converged parallel lines, right?  If you haven't here's an example:

You'll notice that in this diagram there exists a series of parallel lines which seem to converge at a "vanishing point" on the horizon.

This in fact is not the case.  The lines don't actually ever converge, they just look like they do.  It's all a matter of perspective.  Imagine these lines as Cisco's and VMware's virtualization strategies.

Similarly, the notion of forced perspective is a method by which the manipulation of perspective employs an optical illusion to make something appear closer, father away, larger or smaller than it actually is (see the title image above*.)

The announcements from Cisco and VMware are very much like these examples. Whilst they offer excellent opportunities for improving the management and security of virtual infrastructure, it's very much Machiavellian marketing — the end is going to justify the means.

Speaking to either Cisco or VMware you're asked to suspend disbelief and accept that these two companies share a common blueprint for a datacenter OS, but they don't.  In fact, they're quite different, and the balance of who needs whom more is also very lopsided.

Despite the close technical partnership needed to pull off the integration of the Nexus 1000v as the first third party virtual switch (which we've been talking about for almost two years,) Cisco and VMware really are on parallel trajectories in terms of their visions regarding the datacenter OS; how it's designed, provisioned, deployed, managed and governed…and by whom.

Cisco is approaching this primarily as an infrastructure transformation play as a way of clawing back what they lost when the network access layer become absorbed into the virtual hosts while VMware is busy distancing itself from the infrastructure and elevating the discussion to that of the cloud in an effort to stave off Microsoft and Citrix.

Each want to own your datacenter, and while they play nice on the surface, there's really a nasty game of tug of war going on here.  This is a marriage borne of convenience, nothing more.

You try and unify Cisco's DC 3.0 vision with VMware's Virtual Datacenter OS blueprint and tell me how they mesh.

2)  Dear Virtual SysAdmin: You're fired as the network admin.  You're cool with that, right?

It's funny how both Cisco and VMware's marketing folk in the sessions discussing the release of the Nexus 1000v vSwitch, both snarkily (and rhetorically) posited "How many of you Virtual SysAdmins have coordination and communication issues between your virtualization and network teams?"

Leading the witness further, the next question was  "Don't you just hate having to fight to get the network teams to give you a trunk port on an access switch?" 

They followed that up with "Your prayers are answered!  The 1000v will allow you to give the network provisioning back to the network  teams and let them control the networking and connectivity.  Isn't that great?" 

While most nodded away in the affirmative to the first and second questions, I didn't see one audience member who answered positively on the latter.  What makes anyone think the vSysAdmins *want* to give up the control of the virtual networking layer and be at the mercy of the networking teams again?

Interesting battle ground for sure.  Now, please don't misinterpret my commentary as a suggestion that this is a bad thing, but we're already in the middle of a "West Side Story" turf war over organizational fiefdoms.  This will, depending upon what sort of contention exists already, make a really tenuous issue even more so.

3) Software Sucks.  Hardware Rules.   I hope you like ping pong.

I hinted at this point in my post titled (The Network is the Computer…)  The reality is that much like point #1, Cisco could care less in the long term about the Nexus 1000v as a software switch running in someone else's backyard operating environment, but rather introduces it to enable the landscape clawback it gets to enjoy in the short term and make relevant once again it's big network iron in the longer timeframe.

A telling slide was the announcement of what's coming AFTER the Nexus 1000v in one of the sessions that I have not seen presented in detail elsewhere — that is Cisco's goal to extract networking out of the host completely.

The plan as discussed is to utilize what Cisco calls an "initiator" to replace the 1000v and force traffic, after specialized tagging which denotes affinity of flows to specific VM ID's, and ship them straight back out the network interfaces to a waiting Cisco 5000/7000 switch for processing.  Hence the ping-pong mention above.

Sorry for the quality of the picture as I took it sitting behind somebody, but here's a slide denoting just this very thing:

 
The notion of a third party switching capability is really just a way for Cisco to push the access layer back to where they think it rightfully belongs — in the physical switch.

Cisco claims that VMware and they have submitted this tagging specification to the IEEE for review/ratification.   I find that very interesting.

I wrote about the need for such a technology at both the virtualization layer and more importantly the application/data level in June of 2007. 

Check out my post which described how I suggested Crossbeam do the exact same thing by way of something I called ADAPT (Applied Data and Application Policy Tagging) which describes this very thing.  What's next, they're going to announce vNAC? 😉

—

All in all, the Cisco/VMware relationship is about as natural looking as the Microsoft/Citrix version — it's sort of like a midget dating a six foot supermodel…someone's getting the better end of the footrub in that relationship, too.

So, how about it?  Am I stating the obvious again — and does it need to be stated?

/Hoff

*image from "The Eye of Brad" flickrstream