Rational Survivability

Edward Haletky wrote an interesting piece recently titled "CISecurity Guide to VMware Security Falls Far Short" in which he lays down some well-articulated criticisms of the first CIS benchmark for VMware.

Edward’s primary problem with the benchmark can be summarized well by this paragraph:

While the Benchmark was the first of its kind, it is nothing more than the Linux benchmark with some small changes for VMware ESX. Following these steps will increase security but it is by no means a panacea. Do not let it give you a false sense of security.

I think Edward set his expectations a little high prior to review, as I’m pretty sure the word panacea wasn’t used in the syllabus 😉

I don’t disagree with Edward that the flavor of the benchmark is very much a generic set of guidelines focused primarily on securing the underlying Linux-based service console and basic configuration for overall "system" hardening, but we need to realize a couple of things to keep the benchmark in perspective:

1. The benchmark was the first of its kind.  It’s almost 10 months old!  The second version is underway right now as a matter of fact.
2. In between when the benchmark was released and now, we’ve seen the emergence of the embedded version of VMware and much needs to change to address that.
3. The benchmark was designed to be generic and give virtual system administrators a baseline on basic security hardening, not serve as the end-all, be-all for some mythical security end-state.
4. The challenge for those of us who contributed (as I did) was that we had to keep the document vendor/tool agnostic which makes it difficult to frame solutions.
5. Lots of things have changed.

Keep in mind that this is a "level 1" benchmark whose settings/actions are as follows:

• Can be understood and performed by system administrators with any level of security knowledge and experience;
• Are unlikely to cause an interruption of service to the operating system or the applications that run on it; and
• Can be automatically monitored either by CIS Scoring Tools or by CIS Certified tools available from security software vendors. 

This isn’t about being defensive regarding the benchmark as I’ll agree that we could have done much, much more in terms of providing more meatier substance as it relates to how to better secure the ecosystem of mechanicals that a virtualized environment touches. 

However, the scope of a document that effectively addresses the security concerns across this immense landscape would be a huge undertaking.

One of the other difficulties in creating a guideline like this is the fact that those responsible for securing virtualized environments are not security professionals.  As I’ve spoken about previously, the operational realities of who is managing and securing our virtualized infrastructure is cause for concern.

Thus, when creating a guide like this, it’s best to start with the underlying basics and then branch out from there; involve the network and security teams as required.  As Edward himself wrote in this piece, "Good virtual security requires better IT teamwork," to properly secure your virtualized infrastructure, it’s going to take cooperation and expertise from many camps.    

Edward also has written a book titled "VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers."  Interestingly, I found the security sections weak for many of the same high-level reasons he listed in his review of the CIS benchmark.  Security is most definitely in the eye of the "bookholder." 😉

In the meantime, if you’re interested in some additional security/hardening guides and tools for VMware environments, check out the following:

• VMWare – Updated VI3 Security Hardening Guide (7/8/08)
• DoD/DISA – Security Technical Implementation Guide (STIG)
• NSA SNAC – VMware ESX Server 3 Configuration Guide
• Tripwire – ConfigCheck

/Hoff

Amrit turned me onto a Network World article titled "12 Ways to Visualize Network Security" in which his analog of security as a cheese grater is featured.

Yup, there’s castles and cars and…

In an attempt to annoy the crap out of everyone, I decided to start spewing out my candidates via twitter (beaker) so as to force as many un-follows as possible. 

Here are some of my off-the-cuffs [remember, these have to fit in < 140 characters]:

• Security is like Escargot. It’s crunchy on the outside, chewy on the inside, and like everything else, should be blamed on the French!
• Security is like Kimchee…to make it you have to slap it together, bury it and then dig it up when it smells to explain how special it is..
• Security is like Durian: It’s lousy in airports, stinks when exposed and looks oddly out of place no matter how you slice it…
• Security is like fertilizer, the more shit you spread around the worse it gets and watering it down only makes it worse
• Security is like a vibrator, the more you have to use it, the less fun the real business becomes…
• Security is like weed, homeopathy and faith healing; sometimes nothing beats cutting the tumor out, but faith in snake oils is more fun
• Security is like a pig; well, ’nuff said.
• Security is like your ’82 Ford Escort; you can keep telling everyone that it was your mom’s ride & gets good mileage, but everyone knows…
• Security is like a pomegranate; seriously, who the fuck thought it was a good idea to try THAT!
• Security is like balut; when crunchy on the outside, chewy in the middle doesn’t work, go crunchy everywhere?  Sweet Jesus.
• Security is like a vacuum cleaner; both have dirtbags and "suckage" is the primary metric.

Sadly, nobody un-followed and instead I got like 10 new TwitterBots following me instead.  Ain’t that a bitch?

/Hoff

P.S. My man Mogull flung back some fine satirical smackage…nicely played, sir!:

Yo!  BeanSec! is once again upon us.  Wednesday, July 16th, 2008.

PLEASE NOTE THE VENUE CHANGE BELOW!  THIS WILL BECOME PERMANENT.

BeanSec! is an informal meetup of information security
professionals, researchers and academics in the Greater Boston area
that meets the third Wednesday of each month. 

I say again, BeanSec! is hosted the third Wednesday of every month.  Add it to your calendar.

Come get your grub on.  Lots of good people show up.  Really.

Unlike other meetings, you will not be expected to pay dues, “join
up”, present a zero-day exploit, or defend your dissertation to attend.

Middlesex Lounge: 315 Mass Ave, Cambridge 02139.  We are moving locations due to better seating and the fact that the Enormous Room (our prior location) no longer serves food. ;(

Don’t worry about being "late" because most people just show up when they can. 6:30 is a good time to aim for. We’ll try and save you a seat. There is a plenty of parking around or take the T.

In case you’re wondering, we’re getting about 30 people on average per BeanSec! Weld, 0Day and I have been at this for just almost 2 years and without actually *doing* anything, it’s turned out swell.

The food selection is basically high-end finger-food appetizers and the drinks are really good; an attentive staff and eclectic clientèle make the joint fun for people watching. I’ll generally annoy you into participating somehow, even if it’s just fetching napkins. 😉

See you there.

/Hoff, /0Day, and /Weld

Quis custodiet ipsos custodes? Who will watch the watchers?

Short and sweet and perhaps a grumpy statement of the obvious: Security Analyst Sausage Machine Firms quash innovation in vendors’ development cycles and in many cases prevent the consumer — their customers — from receiving actual solutions to real problems because of the stranglehold they maintain on what defines and categorizes a "solution."

What do I mean?

If you’re a vendor — emerging or established — and create a solution that is fantastic and solves real business problems but doesn’t fit neatly within an existing "quadrant," "cycle," "scope," or "square," you’re SCREWED.  You may sell a handful of your widgets to early adopters, but your product isn’t real unless an analyst says it is and you still have money in the bank after a few years to deliver it.

If you’re a customer, you may never see that product develop and see the light of day and you’re the ones who pay your membership dues to the same analyst firms to advise you on what to do!

I know that we’ve all basically dropped trow and given in to the fact that we’ve got to follow the analyst hazing rituals, but that doesn’t make it right.  It really sucks monkey balls.

What’s funny to me is that we have these huge lawsuits filed against corporations for anti-trust and unfair business practices, and there’s nobody who contests this oligopoly from the sausage machine analysts — except for other former analysts who form their own analyst firms to do battle with their former employers…but in a kindler, gentler, "advisory" capacity, of course…

Speaking of which, some of these folks who lead these practices often times have never used, deployed, tested, or sometimes even seen the products they take money for and advise their clients on.  Oh, and objectivity?  Yeah, right.  If an analyst doesn’t like your idea, your product, your philosophy, your choice in clothing or you, you’re done.

This crappy system stifles innovation, it grinds real solutions into the dirt such that small startups that really could be "the next big thing" often are now forced to be born as seed technology starters for larger companies to buy for M&A pennies so they can slow-roll the IP into the roadmaps over a long time and smooth the curve once markets are "mature."

Guess who defines them as being "mature?"  Right.

Crossing the chasm?  Reaching the tipping point?  How much of that even matters anymore?

Ah, the innovator’s dilemma…

If you have a product that well and truly does X, Y and Z, where X is a feature that conforms and fits into a defined category but Y and Z — while truly differentiating and powerful — do not, you’re forced to focus on, develop around and hype X, label your product as being X, and not invest as much in Y and Z.

If you miss the market timing and can’t afford to schmooze effectively and don’t look forward enough with a business model that allows for flexibility, you may make the world’s best X, but when X commoditizes and Y and Z are now the hottest "new" square, chances are you won’t matter anymore, even if you’ve had it for years.

The product managers, marketing directors and salesfolk are forced to
fit a product within an analyst’s arbitrary product definition or risk
not getting traction, miss competitive analysis/comparisons or even get
funding; ever try to convince a VC that they should fund you when
you’re the "only one" in the space and there’s no analyst recognition
of a "market?"

Yech.

A vendor’s excellent solution can simply wither and die on the vine in
a battle of market definition attrition because the vendor is forced to
conform and neuter a product in order to make a buck and can’t actually
differentiate or focus on the things that truly make it a better
solution.

Who wins here? 

Not the vendors.  Not the customers. The analysts do. 

The vendor pays them a shitload of kowtowing and money for the privilege to show up in a box so they get recognized — and not necessarily for the things that truly matter — until the same analyst changes his/her mind and recognizes that perhaps Y and Z are "real" or creates category W, and the vicious cycle starts anew.

So while you’re a vendor struggling to make a great solution or a customer trying to solve real business problems, who watches the watchers?

/Hoff

The InterWeb’s broken!
Oy, vadda mess!
Kaminsky tells all
Patch your damned DNS!

VMware’s Greene has gone virtual,
where will she land?
Maritz is the new boss,
since Diane got canned

Speaking of virtual
Ballmer’s jumpin’ with glee,
for twenty-eight bucks
you can own Hyper-V!

Oh the Senate just gave us
a shitty surprise-a,
those spineless rat bastards
just re-voted in FISA

Hear that sound in the background?
That’s the ACLU crying
The telcos and Intel
get rewarded for spying!

That’s right they can wiretap
your comms with impunity
Our elected officials
just gave them immunity!

The new iPhone this Friday,
faster speeds, GPS
If only they’d fix
AT&T’s coverage mess

Poor Jerry Yang
and his Yahoo-stacked board
If Carl gets his way
Yang will fall on his sword

Matasano’s first product
took a while to cook
Many firewalls?  Hard to Manage?
Give Playbook a look.

As a wrap-up this time
I’ll pull the guilt lever
Read this post on my charity
and donate to Kiva!

"…everyone who wants to make a difference should just go ahead and get
their own foreign policy and stop waiting on change from above."– Thomas Barnett

Inspired by my friend Gunnar Peterson, I’ve committed to begin funding Kiva Micro-loans in the next 30 days with a goal to fund up to $1,000 by year end.

What does Kiva do and what is a micro-loan?

Kiva is focused on serving the working poor

Kiva’s mission is to connect people through lending for the sake of alleviating poverty.

Kiva is the world’s first person-to-person micro-lending website,
empowering individuals to lend directly to unique entrepreneurs in the
developing world. The people you see on Kiva’s site are real
individuals in need of funding – not marketing material.

When you browse entrepreneurs’ profiles on the site, choose
someone to lend to, and then make a loan, you are helping a real person
make great strides towards economic independence and improve life for
themselves, their family, and their community. Throughout the course of
the loan (usually 6-12 months), you can receive email journal updates
and track repayments. Then, when you get your loan money back, you can
relend to someone else in need.

Here’s a snippet from Gunnar’s posting which describes his experience with Kiva:

About a year ago, we signed up for Kiva, which is a microlender. One of our first loans went to Sith Saron, who lives in Siem Reap Province in Cambodia. She needed a $1,000 for a cow, seeds, and a motorcycle for her farm.

Sith Saron is 37 years old and the mother of 7 children. She sells Khmer traditional cakes such as Num Korm, Num Bot, and Num Krouk to the people in her community and usually earns up to $4 each day. Her husband, meanwhile, works in his rice paddy growing crops as well as several kinds of vegetables. Two of her children are employed at a hotel, but the others are students.

The loan had a 18 month pay back date, and just a couple of weeks ago (about 10 months after taking out the loan), she paid the loan in full

If you are interested in helping me — and thus others — with contributing to the micro-loan movement, either sign-up to donate directly yourself, or feel free to donate via gift certificate to my pool and we can make an even bigger difference!

If you want to send a Kiva certificate, you can do so through the PayPal-enabled link above and use my email addy as the target recipient: choff [@] packetfilter.com

At my birthday BBQ bash this weekend, in lieu of gifts I’ve asked for folks to donate to my pool for this year to fund multiple loans.

My family of three young girls and my lovely wife are all very excited about being able to participate in this process both domestically and internationally. 

In fact, all three of my kids are invested in giving up material goods and gifts in exchange for donations to Kiva.  How cool is that? 

Thanks to Gunnar again for the motivation and Thomas Barnett for his inspiring words.

/Hoff

Update: Within 3 minutes of posting this, my bud Zach already donated!  Fantastic!

Alessandro from virtualization.info and I were chatting today regarding VMware’s latest best-practices document titled "DMZ Virtualization with VMware Infrastructure." 

This is a nine page overview that does a reasonably good job of laying out many of the architectural/topological options available when thinking about taking the steps toward virtualizing what some consider the "final frontier" in the proving grounds of production-level virtualization — the (Internet-facing) DMZ.

The whitepaper was timely because I was just finishing up my presentation for Blackhat and was busy creating a similar set of high-level architectural examples to use in my presentation.  I decided to reference those in the document because they quite elegantly represent the starting points that many folks would use as a stepping off point in their virtual DMZ adventures.

…and I think it will be an adventure punctuated perhaps by evolutionary steps as documented in the options presented in the whitepaper.

As I read through the document, I had to remind myself of the fact that this was intended to be a high-level document and not designed to cover the hairy edges of network and security design. 

The whitepaper highlighted some of the reasonable trade-off’s in complexity, resiliency, management, functionality, operational expertise, and cost but given where my head and focus are today, I have to admit that it still gnawed at me from a security perspective which is still too weak for my liking.

I’ve hinted at why in my original Four Horsemen slide, and I’m going to be speaking for 75 minutes on the topic at Blackhat, so come get your VirtSec boogie on there for a full explanation…

Alessandro got dinged in a comment on his blog for a statement in which he suggested that partially-collapsed as well as fully-collapsed DMZ’s with virtual separation of trust zones "…should be avoided at all costs because they imply the inviolability of the hypervisor (at any level: from the virtual networking to the kernel) something that nor VMware neither any other virtualization vendor can grant."

This appears contradictory to his initial assessment of DMZ virtualization wherein he stated that "…there [is] nothing bad in virtualizing the DMZ as long as we are fully aware of the risks."  In a way, I think I understand exactly where Alessandro is coming from, even if I don’t completely agree with him (or at least I partially do…)

This really paints an altogether unfortunate and yet accurate picture of the circular arguments folks engage in when they combine the following topics in a single argument:

• Securing virtualization
• Virtualizing security
• Security via virtualization

In the same way that we trust our operating system vendors who provide us with the operational underpinnings of our datacenters with the hope that they will approach a reasonable level of "security" in their products, we are basically at the same point with our virtualization (OS) platform providers.

Hope is not a strategy, but it seems we’ve at least accepted it for the time being… ;(

Sure there are new attack vectors and operational risks, but the slippery slope of not being able to really quantify whether you are more or less at risk based solely on the one-dimensional data point of the infallibility of the hypervisor  and then write the whole concept off seems a little odd to me.

If you’re truly assessing risk in the potential virtualization of your DMZ, you’ll take the operational/architectural guidelines as well as the subjective business impacts into consideration.  Simply stating that one should or should not virtualize a DMZ without a holistic approach is myopic.

To circle back on the topic, the choice of whether to — and how to — virtualize your DMZ  is really starting to gain traction.  I think the whitepaper took a decent first-pass stab at exploring how one might approach it, but the devil’s in the details — or at least the devil’s 4 horsemen are 😉

/Hoff

Today was the deadline for submission for all selected Blackhat presentations. 

I’m giving a 75 minute talk titled "The Four Horsemen of the Virtualization Apocalypse" which is based upon my original blog posting here.

I dutifully uploaded my presentation to Ping and the gang at Blackhat HQ today (on time, that’s a first!) with a sigh of relief and accomplishment.  I’ve done hundreds of presentations over the years, but this one is special.

I have to say that I poured my heart and soul into this presentation.  I went all "Zen and the Art of Presentation" for most of it and I think that combined with the dozens of hours I put into the content, the diagrams and animations turned out purdy. 😉

Once BH is done, I’ll be posting it online with my narrative as I have my other presentations.

This cathartic little post is just the final little exhale of this project prior to numerous advance rehearsals, the first of which I will be inflicting upon my unwitting guests (75+ of them thus far) at my July 5th Pig Roast & Mojito festival in honor of another notch in the annual belt I’ve managed to stay alive on this hunk o’ rock.

Speaking of which, if you’re in the MA area and want an amazing cuban or southern-style pulled pork feast with all the trimmings, drop me a line as everyone’s welcome…many of the BeanSec’rs are coming, you should too!

Happy 4th/5th!

/Hoff

Thanks to Alan Shimel and his pre-Blackhat Security Bloggers Network commentary, a bunch of interesting folks are commenting on the topic of virtualization security (VirtSec) which is the focus of my preso at Blackhat this year.

Mike Rothman did his part this morning by writing up a thought-provoking piece opining on the lack of a near-term market for VirtSec solutions:

So I’m not going to talk about technical stuff. Yet, I do feel compelled to draw the conclusion that despite the dangers, it doesn’t matter. All the folks that are trying to make VirtSec into a market are basically just pushing on a rope.

That’s right. Now matter how hard you push (or how many blog postings you write), you are not going to make VirtSec into a market for at least 2 years. And that is being pretty optimistic. So for all those VCs that are thinking they’ve jumped onto the next big security opportunity, I hope your partnership will allow you to be patient.

Again, it’s not because the risks of virtualization aren’t real. If guys like Hoff and Thomas say they are, then I tend to believe them. But Mr. Market doesn’t care what smart guys say. Mr. Market cares about budget cycles and priorities and political affiliations, and none of these lead me to believe that VirtSec revenues are going to accelerate anytime soon.

Firstly, almost all markets take a couple of years to fully develop and mature and VirtSec is no different.  Nobody said that VirtSec will violate the laws of physics, but it’s also a very hot topic and consumers/adopters are recognizing that security is a piece of the puzzle that is missing.

In many cases this is because virtualization platform providers have simply marketed virtualization as being "as secure" or "more secure" than than their physical counterparts.  This, combined with the rapid adoption of virtualization, has caused a knee jerk reactive reaction.

By the way, this is completely par for the course in our industry.  If you act surprised, you deserve an Emmy 😉

Secondly, and most importantly to me, Mike did me a bit of a disservice by intimating that my pushing the issues regarding VirtSec are focused solely on the technical.  Sadly, that’s so far off base from my "fair and balanced" perspective on the matter because along with the technical issues, I constantly drum home the following:

• The biggest challenge we have with virtualization in the short term (the next couple of years — the same timeframe Mike suggests VirtSec will take to percolate) is operational and organizational, and not technical.

  You can find recent postings on that topic in posts such as Security Pros Say VirtSec Is An Operations Problem? and The Challenge of Virtualization Security: Organizational and Operational, NOT Technical   

• Along with the organizational and operational issues comes the need for visibility and visualization of what is going on within these virtual environments.

  You can find my latest entry on this (posted early today, actually) in my entry titled Visualization Through Virtualization…

"Nobody Puts Baby In the Corner"

Painting only one of the legs of the stool as my sole argument isn’t accurate and doesn’t portray what I have been talking about for some time — and agree with Mike about — that these challenges are more than one-dimensional.

The reality is that Mike is right — the budget, priority and politics will bracket VirtSec’s adoption, but only if you think of VirtSec as a technical problem.

Is VirtSec a market?  My opinion: it’s an instantiation of technology, practice and operational adjustment brought forth as a derivative of a disruptive technology and prevailing market conditions. 

Does that mean it’s a feature as opposed to a market?  No.  In my opinion, it’s an evolution of an existing market, rife with existing solutions and punctuated by emerging ones.

The next stop is how "security" will evolve from VirtSec to CloudSec…

/Hoff

Check out the update below…

Were I in the UTM business, I’d be engaging the reality distortion field and speed-dialing my patent attorneys at this point.

Fortinet has recently had some very interesting patent applications granted by the PTO.

Integrated network and application security, together with virtualization technologies, offer a powerful and synergistic approach for defending against an increasingly dangerous cyber-criminal environment. In combination with its extensive patent-pending applications and patents already granted, Fortinet’s newest patents address critical technologies that enable comprehensive network protection:

• U.S. Patent #7,333,430 – Systems and Methods for Passing Network Traffic Data – directed to efficiently processing network traffic data to facilitate policy enforcement, including content scanning, source/destination verification, virus scanning, content detection and intrusion detection;

• U.S. Patent #7,340,535 – System and Method for Controlling Routing in a Virtual Router System – directed to controlling the routing of network data, and providing efficient configuration of routing functionality and optimized use of available resources by applying functions to data packets in a virtual environment;

• U.S. Patent #7,376,125 – Service Processing Switch – directed to providing IP services and IP packet processing in a virtual router-based system using IP flow caches, virtual routing engines, virtual services engines and advanced security engines;

• U.S. Patent # 7,389,358 – Distributed Virtual System to Support Managed, Network-based Services – directed to a virtual routing system, which includes processing elements to manage and optimize IP traffic, useful for service provider switching functions at Internet point-of-presence (POP) locations.

These patents could have some potentially profound impact on vendors who offer "integrated security" by allowing for virtualized application of network security policy.  These patents could easily be enforced outside of the typically-defined UTM offerings, also.

I’m quite certain Cisco and Juniper are taking note as should be anyone in the business of offering virtualized routing/switching combined with security — that’s certainly a broad swath, eh?

On a wider note, I’ve actually been quite impressed with the IP portfolio that Fortinet has been assembling over the last couple of years.  If you’ve been paying attention, you will notice (for example) that that they have scooped up much of the remaining CoSine IP as well as recently acquired IPlocks’ database security portfolio.

If I were they, the next thing I’d look for (and would have a while ago) is to scoop up a Web Application Firewall/Proxy vendor…

I trust you can figure out why…why not hazard a guess in the comments?

/Hoff

Updated:  It occured to me that this may be much more far-reaching than just UTM vendors, that basically this could affect folks like Crossbeam, Check Point, StillSecure, Cisco, Juniper, Secure Computing, f5…basically anyone who sells a product that mixes the application of security policy with virtualized routing/switching capabilities…

Now, I’m not a lawyer, I just play one on teh Interwebs.