Rational Survivability

Short and sweet…

With the continued network abstraction and “simplicity” presented by public cloud platforms like AWS EC2* wherein instances are singly-homed and the level of networking is so dumbed down so as to make deep networking knowledge “unnecessary,” will the skill sets of next generation operators become “network stupid?”

The platform operators will continue to hire skilled network architects, engineers and operators, but the ultimate consumers of these services are being sold on the fact that they won’t have to and in many cases this means that “networking” as a discipline may face a skills shortage.

The interesting implications here is that with all this abstraction and opaque stacks, resilient design is still dependent upon so much “networking” — although much of it is layer 4 and above.  Yep, it’s still TCP/IP, but the implications that the dumbing down of the stack will be profound, especially if one recognizes that ultimately these Public clouds will interconnect to Private clouds, and the two networking models are profoundly differentiated.

…think VMware versus AWS EC2…or check out the meet-in-the-middle approach with OpenStack and Quantum…

I’m concerned that we’re still so bifurcated in our discussions of networking and the Cloud.

One the one hand we’re yapping at one another about stretched L2 domains, fabrics and control/data plane separation or staring into the abyss of L7 proxies and DPI…all the while the implications of SDN and emergence of new protocols, the majority of which are irrelevant to the consumers deploying VMs and apps atop IaaS and PaaS (not to mention SaaS,) makes these discussions seem silly.

On the other hand, DevOps/NoOps folks push their code to platforms that rely less and less on needing to understand or care how the underlying “network” works.

Its’ hard to tell whether “networking” in the pure sense will be important in the long term.

Or as Kaminsky so (per usual) elegantly summarized:

What are your thoughts?

/Hoff

*…and yet we see more “complex” capabilities emerging in scenarios such as AWS VPC…

Related articles

• Cloud Bursting: Gateway Drug for Hybrid Cloud (devcentral.f5.com)
• QuickQuip: “Networking Doesn’t Need a VMWare” (rationalsurvivability.com)

Cloud Computing Image (Photo credit: Wikipedia)

If you’ve been paying attention to the rash of security startups entering the market today, you will no doubt notice the theme wherein the majority of them are, from the get-go, organizing around deployment models which operate from “The Cloud.”

We can argue that “Security as a service” usually refers to security services provided by a third party using the SaaS (software as a service) model, but there’s a compelling set of capabilities that enables companies large and small to be both effective, efficient and cost-manageable as we embrace the “new” world of highly distributed applications, content and communications (cloud and mobility combined.)

As with virtualization, when one discusses “security” and “cloud computing,” any of the three perspectives often are conflated (from my post “Security: In the Cloud, For the Cloud & By the Cloud…“):

In the same way that I differentiated “Virtualizing Security, Securing Virtualization and Security via Virtualization” in my Four Horsemen presentation, I ask people to consider these three models when discussing security and Cloud:

1. In the Cloud: Security (products, solutions, technology) instantiated as an operational capability deployed within Cloud Computing environments (up/down the stack.) Think virtualized firewalls, IDP, AV, DLP, DoS/DDoS, IAM, etc.
2. For the Cloud: Security services that are specifically targeted toward securing OTHER Cloud Computing services, delivered by Cloud Computing providers (see next entry) . Think cloud-based Anti-spam, DDoS, DLP, WAF, etc.
3. By the Cloud: Security services delivered by Cloud Computing services which are used by providers in option #2 which often rely on those features described in option #1.  Think, well…basically any service these days that brand themselves as Cloud…

What I’m talking about here is really item #3; security “by the cloud,” wherein these services utilize any cloud-based platform (SaaS, PaaS or IaaS) to delivery security capabilities on behalf of the provider or ultimate consumer of services.

For the SMB/SME/Branch, one can expect a hybrid model of on-premises physical (multi-function) devices that also incorporate some sort of redirect or offload to these cloud-based services. Frankly, the same model works for the larger enterprise but in many cases regulatory issues of privacy/IP concerns arise.  This is where the capability of both “private” (or dedicated) versions of these services are requested (either on-premises or off, but dedicated.)

Service providers see a large opportunity to finally deliver value-added, scaleable and revenue-generating security services atop what they offer today.  This is the realized vision of the long-awaited “clean pipes” and “secure hosting” capabilities.  See this post from 2007 “Clean Pipes – Less Sewerage or More Potable Water?”

If you haven’t noticed your service providers dipping their toes here, you certainly have seen startups (and larger security players) do so.  Here are just a few examples:

• Qualys
• Trend Micro
• Symantec
• Cisco (Ironport/ScanSafe)
• Juniper
• CloudFlare
• ZScaler
• Incapsula
• Dome9
• CloudPassage
• Porticor
• …and many more

As many vendors “virtualize” their offers and start to realize that through basic networking, APIs, service chaining, traffic steering and security intelligence/analytics, these solutions become more scaleable, leveragable and interoperable, the services you’ll be able to consume will also increase…and they will become more application and information-centric in nature.

Again, this doesn’t mean the disappearance of on-premises or host-based security capabilities, but you should expect the cloud (and it’s derivative offshoots like Big Data) to deliver some really awesome hybrid security capabilities that make your life easier.  Rich Mogull (@rmogull) and I gave about 20 examples of this in our “Grilling Cloudicorns: Mythical CloudSec Tools You Can Use Today” at RSA last month.

Get ready because while security folks often eye “The Cloud” suspiciously, it also offers up a set of emerging solutions that will undoubtedly allow for more efficient, effective and affordable security capabilities that will allow us to focus more on the things that matter.

/Hoff

Related articles by Zemanta

• [Webinar] Cloud Based Security Services: Saving Cloud Computing Users From Evil-Doers (rationalsurvivability.com)
• The Four Horsemen Of the Virtualization (and Cloud) Security Apocalypse… (rationalsurvivability.com)
• DDoS – A Moose On Cloud’s Table Or A Pea Under The Mattress? (rationalsurvivability.com)
• Six Year Old Rationalizes the Cloud (rationalsurvivability.com)
• Cloud Computing Security: (Orchestral) Maneuvers In the Dark? (rationalsurvivability.com)
• Cloudifornication: Indiscriminate Information Intercourse Involving Internet Infrastructure (rationalsurvivability.com)
• You Can’t Secure The Cloud… (rationalsurvivability.com)
• Security and the Cloud – What Does That Even Mean? (rationalsurvivability.com)

• Building/Bolting Security In/On – A Pox On the Audit Paradox! (rationalsurvivability.com)
• Security As A Service: Virtual Patching for Cloud Environments (canadacloud.biz)
• Security As A Service Is More Than A Virtual Security Gateway (securecloudreview.com)
• The Inevitability of Security-as-a-Service (securecloudreview.com)

My buddy Bill Brenner (@billbrenner70) blogged a question that stemmed from a “discussion” I seem to have initiated yesterday: “Do People In Security Blog Too Much?”

He was kind enough to accommodate a clarification from me in which I reiterated that my chief complaint regarding excessive self-promotion by individuals  was “not about volume, but variety.”

To be clear, RT’ing a link (however modified) that is clearly designed to self-promote onesself is, in my opinion, bordering on SPAM-like behavior when one does it 10+ times in a 24 hour period.

I don’t mind a lot of tweets.  I mind a lot of the same tweets.

…The same way people get annoyed with folks who live tweet conferences, I suppose.

Now, people have the right to tweet whatever they like, as often as they like, but the reason I brought this up was because I was truly interested in whether or not the individual in question understood the impact/annoyance it caused.

Based on his reply, the “data he had to suggest ‘increased engagement,’ and what was clearly a strategy behind this activity, it became apparent he didn’t.

So I did what anyone in my position has the option to do: I unfollowed.  This was followed by an additional comment from the author that only “…~0.1% of followers had a negative response” to his RT’ing [approximately 5/4200 people.]

I found that odd, since I had at least 10 DM’s in my mailbox from followers who reacted to my tweets surrounding this issue.

5 or so others then piped up suggesting they were also annoyed but, like me, had not said anything.

As I mentioned, I wasn’t looking for anything like an apology — it’s not my place to, nor am I arrogant enough to suggest I’m owed one — but I did want him to understand that there were ramifications that either he was unaware of or simply ignoring.  Again, his choice.

I probably *do* tweet too much for many people’s likes — and they unfollow accordingly.  However, I operate under the “code” that I try very hard to not RT anything self-promotional more than TWICE in a 24 hour period.  I figure that with timezone deltas, but with RSS feeds and other RT’s from interested parties, that’s sufficient.

Am I potentially missing people?  Sure.  But the way I look at it is that if it’s interesting enough, people will find it.

I’m not in the “business” of “SEO for Twitter” (h/t to @SecureTom for the phrase,) but that’s a personal choice.

I will suggest, however, that people are smarter than many give them credit for — you can get cute and change the preamble, but if you deluge their timeline with self-promotion, expect them to one day get grumpy enough to find the unfollow button…and use it.

/Hoff

Here they are…*some* of my favorite Tweets O’ the Week that I curated:

• Unless you like fish, stop chasing red herrings.
• The hypervisor is/should be the least of your security concerns in a virtualized environment. The ops & mgmt layer should be
• The next 1 of you (us) who starts whining about how broken our industry is without doing anything about it gets posted to the hamster wall
• This is the new norm I call anti-FUD FUD: security vendors shitting where they eat in an (em)pathetic attempt to gain cred. How ’bout fixin?
• Congrats on $60MM funding @appirio. It’s great u’ll be able to afford to create even more BS marketing contests you rig the outcome to ;p
• Protip: The state of the Security Industry always looks like shit in the middle of a “breaker” hacker con.  By design. You’re welcome.
• More negativity, navel gazing & security apocalypse hype. Funny how “experts” doing the sky-is-falling chicken dance never propose solutions
• Awkward moment today: someone presenting me slides re: Cloud Security that I built on an initiative I created and a group I lead. o_O
• Oh! Right! Cloud security, visibility & transparency. Why didn’t I think of that?!
• North by Northwest is basically the Hitchcock version of Anonymous, Wikileaks…with biplanes and better acting.
• I will soon utilize HTTPS/SSL to encrypt all my tweets. Those of you who are not Beaker Certified will be unable to decipher my madness
• Out of complete ignorance: is SXSW like Burning Man for nerds who only discuss things that are battery operated?
• What a bunch of chicken shits. 20 DM’s later and 18 of you vote @MikD as the Ryan Seacrest of Infosec. Like that’s a bad thing?
• My twitter follower count goal is 90210 – that way I can claim I am the Tiffany Amber Theisen of Twitter. It’s the little things…
• Single best way to get uninvited back to weekly meetings is introduce the fact that the host’s model construct for an argument is flawed.
• Oh $gawd. What a bunch of cockblocking going on with respect to $openwashing & who started what. Sigh. #getonwithitalready
• I just sent the most awesome f’ing internal email ever.  If there was EVER a reason for REPLY-ALL, *this* would be it. GRAB YOUR RED STAPLER

Did I miss any? 😉

Update 030712: I’m going to follow this post up with yet another post mortem that includes lessons learned and more details as I can supply them.  I will point out two things:

1. It’s pretty clear that the secondary/tertiary stages of this infestation which led to multiple alerts from my readers is related to the massive WordPress attack you can read about here.  It’s important to note, however, that the first incident (which was chalked up impoperly to a false positive) and a second started with similar symptoms back in late July.  I simply didn’t have the data to correlate.  They were different variants.
2. The support from vendors and the security community has been outstanding.  People with no vested interest in the health of my personal blog have gone out of their way to help, even my hosting provider, Dreamhost (although we got off on a rather rocky footing 😉

I do owe both FireEye (who spotted the original attack) and Dreamhost additional data which I will attempt to retrieve.  I also owe Rich Mogull an apology/explanation regarding why I didn’t immediately take the blog down, risking further infection — I legitimately thought we’d fixed it, but because of the stealth of the malware, I was wrong.  Once I realized I couldn’t contain/isolate it, I did take it down…and then wiped the entire blog/database.

At any rate, thanks for bearing with me though this.  It’s been invaluable to me and I hope you found some value in all of this.

It certainly was interesting and gave me some unique insight into the psychology, behavior, biases and opinions of the community/industry that I didn’t fully appreciate prior.

—

This is an update that I originally included with the post describing the malicious infestation of malware on my WordPress site here.  I’ve split it out for clarity.

The last 12 hours or so have been fun. I’ve had many other folks join in and try to help isolate and eradicate the malware that plagued my WordPress install (read the original post below.)

I was able to determine that the Dreamhost password compromise in January (correlated against logs) was responsible for the (likely) automated injection of malicious PHP code into a plug-in directory that had poor permissions.  This code was BASE64 encoded. It was hard to find.

Further, as was alluded to in my earlier version of this post, the malware itself was adaptive and would only try (based on UA and originating IP) to drop it’s Windows-based trojan executable ONCE by way of a hidden iFrame. Hit it again and you’d never see it.

It was a variant of the Blackhole Exploit kit.

If you ran any up-to-date AV solution (as evidenced by the 6 different brands that people reported,) visiting my site immediately tripped an alert.  I run a Mac and up until today didn’t have such a tool installed. I clearly do now as a detective capability.  This was a silly thing NOT to do as it costs basically nothing to do so these days.

When I made a backup of the entire directory, my VPS hosting provider THEN decided to run a security scan on the directory (serendipity) and notified me via email that it found the malware in the directory 🙁 Thanks.  Great timing.  The funny thing was that all the activity last night and uploaded telemetry must have set something off in Google because only late last night — 30+ days later — did Google flag the site as potentially compromised.  Sigh.

At any rate, I ended up nuking my entire WordPress and mySQL installations and doing a fresh install. I’ve rid myself of almost every plug-in and gone back to a basic theme.  I’ve installed a couple of other detective and preventative tools on the site and will likely end up finally putting the site behind CloudFlare for an additional layer of protection.

Really, I should have done this stuff LONG ago…this was my personal failure.  I owe it to the kindness and attentiveness of those who alerted me to the fact that their AV sensors tripped.

The interesting note is that most of the security pros I know who run Macs and have visited my site in the last 30 days never knew I was infected.  If this were a Mac-targeted malware, perhaps they may have been infected.  The point is that while I’m glad it didn’t/couldn’t infect Mac users, I do care that I could have harmed users with other operating systems.

Further, the “ignorance is bliss” approach is personally alarming to me; without a tool which many security pros sleight as “useless,” I would never have know I was infected.

If anything, it should make you think…

(I originally pre-pended to this post a lengthy update based on my findings and incident response, but per a suggestion from @jeremiahg, I’ve created a separate post here for clarity)

Earlier today I wrote about the trending meme in the blogosphere/security bellybutton squad wherein the notion that security — or the perceived lacking thereof — is losing the “war.”

My response was that the expectations and methodology by which we measure success or failure is arbitrary and grossly inaccurate.  Furthermore, I suggest that the solutions we have at our disposal are geared toward solving short-term problems designed to generate revenue for vendors and solve point-specific problems based on prevailing threats and the appetite to combat them.

As a corollary, if you reduce this down to the basics, the tools we have at our disposal that we decry as useless often times work just fine…if you actually use them.

For most of us, we do what we can to provide appropriate layers of defense where possible but our adversaries are crafty and in many cases more skilled.  For some, this means our efforts are a lost cause but the reality is that often times good enough is good enough…until it isn’t.

Like it wasn’t today.

Let me paint you a picture.

A few days ago a Wired story titled “Is antivirus a waste of money?” hit the wires that quoted many (of my friends) as saying that security professionals don’t run antivirus.  There were discussions about efficacy, performance and usefulness. Many of the folks quoted in that article also run Macs.  There was some interesting banter on Twitter also.

If we rewind a few weeks, I was contacted by two people a few days apart, one running a FireEye network-based anti-malware solution and another running a mainstream host-based anti-virus solution.

Both of these people let me know that their solutions detected and blocked a Javascript-based redirection attempt from my blog which runs a self-hosted WordPress installation.

I pawed through my blog’s PHP code, turned off almost every plug-in, ran the exploit scanner…all the while unable to reproduce the behavior on my Mac or within a fresh Windows 7 VM.

The FireEye report ultimately was reported back as a false positive while the host-based AV solution couldn’t be reproduced, either.

Fast forward to today and after I wrote the blog “You know what’s dead? Security…” I had a huge number of click-throughs from my tweet.

The point of my blog was that security isn’t dead and we aren’t so grossly failing but rather suffering a death from a thousand cuts.  However, while we’ve got a ton of band-aids, it doesn’t make it any less painful.

Speaking of pain, almost immediately upon posting the tweet, I received reports from 5-6 people indicating their AV solutions detected an attempted malicious code execution, specifically a Javascript redirector.

This behavior was commensurate with the prior “sightings” and so with the help of @innismir and @chort0, I set about trying to reproduce the event.

@chort0 found that a hidden iFrame was redirecting to a site hosting in Belize (screen caps later) that ultimately linked to other sites in Russia and produced a delightful greeting which said “Gotcha!” after attempting to drop an executable.

Again, I was unable to duplicate and it seemed that once loaded, the iFrame and file dropper did not reappear.  @innismir didn’t get the iFrame but grabbed the dropped file.

This led to further investigation that it was likely this was an embedded compromise within the theme I was using.  @innismir found that the Sakura theme included “…woo-tumblog [which] uses a old version of TimThumb, which has a hole in it.”

I switched back to a basic built-in theme and turned off the remainder of the non-critical plug-ins.

Since I have no way of replicating the initial drop attempt, I can only hope that this exercise which involved some basic AV tools, some browser debug tools, some PCAP network traces and good ole investigation from three security wonks has paid off…

ONLY YOU CAN PREVENT MALWARE FIRES (so please let me know if you see an indication of an attempted malware infection.)

Now, back to the point at hand…I would never have noticed this (or more specifically others wouldn’t) had they not been running AV.

So while many look at these imperfect tools as a failure because they don’t detect/prevent all attacks, imagine how many more people I may have unwittingly infected accidentally.

Irony?  Perhaps, but what happened following the notification gives me more hope (in the combination of people, community and technology) than contempt for our gaps as an industry.

I plan to augment this post with more details and a conclusion about what I might have done differently once I have a moment to digest what we’ve done and try and confirm if it’s indeed repaired.  I hope it’s gone for good.

Thanks again to those of you who notified me of the anomalous behavior.

What’s scary is how many of you didn’t.

Is security “losing?”

Ask me in the morning…I’ll likely answer that from my perspective, no, but it’s one little battle at a time that matters.

/Hoff

Related articles

• WordPress Hacked, Security Steps (Astroarch)
• Security professionals DO use anti-virus (eset.com)
• Report: malware pushed by affiliate networks remains the primary growth factor of the cybercrime ecosystem (zdnet.com)
• A Blogger’s Worst Nightmare – Your site has been hacked by malware! (bpwebnews.com)
• You Know What’s Dead? Security… (rationalsurvivability.com)

…well, it is if you listen to many of the folks who spend their time trawling about security conferences, writing blogs (like this one) or on podcasts, it is.  I don’t share that opinion, however.

Lately there’s been a noisy upswing in the security echo chamber of people who suggest that  given the visibility, scope, oft-quoted financial impact and reputational damage of recent breaches, that “security is losing.”

{…losing it’s mind, perhaps…}

What’s troubling about all this hen pecking is that with each complaint about the sorry state of the security “industry,” there’s rarely ever offered a useful solution that is appropriately adoptable within a reasonable timeframe, that satisfies a business condition, and result in an outcome that moves the needle to the “winning” side of the meter.

I was asked by Martin Mckeay (@mckeay) in a debate on Twitter, in which I framed the points above, if “…[I] don’t see all the recent breaches as evidence that we’re losing…that so many companies compromised as proof [that we’re losing.]”

My answer was a succinct “no.”

What these breaches indicate is the constant innovation we see from attackers, the fact that companies are disclosing said breaches and the relative high-value targets admitting such.  We’re also seeing the better organization of advanced adversaries whose tactics and goals aren’t always aligned with the profiles of “hackers” we see in the movies.

That means our solutions aren’t aligned to the problems we think we have nor the motivation and tactics of the attackers that these solutions are designed to prevent.

The dynamic tension between “us” and “them” is always cyclical in terms of the perception of who is “winning” versus “losing.”  Always has been, always will be.  Anyone who doesn’t recognize patterns in this industry is either:

1. New
2. Ignorant
3. Selling you something
4. …or all of the above

Most importantly, it’s really, really important to recognize that the security “industry” is in business to accomplish one goal:

Make money.

It’s not a charity.  It’s not a cause.  It’s not a club.  It’s a business.

The security industry — established behemoths and startups alike — are in the business of being in business.  They may be staffed by passionate, idealistic and caring individuals, but those individuals enjoy paying their mortgages.

These companies also provide solutions that aren’t always ready from the perspective of market, economics, culture, adoptability, scope/impact of problem, etc.  This is why I show the Security Hamster Sine Wave of Pain and why security, much like bell bottoms, comes back into vogue in cycles…generally when those items above converge.

Now, if you overlay what I just said with the velocity and variety of innovation without constraint that attackers play with and you have a clearer picture of why we are where we are.

Of course, no rant like this would be complete without the anecdotal handwaving bemoaning flawed trust models and technology, insecure applications and those pesky users…sigh.

The reality is that if we (as operators) are constrained to passive defense and are expected to score progress in terms of moving the defensive line forward versus holding ground, albeit with collateral damage, then yes…we’re losing.

If, rather, we assess our ability to influence outcomes such that the business can function at an acceptable level of risk, where “winning” and “losing” aren’t measured in emotional baggage or absolutes, then perhaps more often than not, we’d be winning instead of whining.

It’s all a matter of perspective, really.

I think staring at things other than one’s bellybutton can deliver some.

Try it.  It won’t hurt.  Promise.

/Hoff

Related articles

• Bringing Sexy back (to Security): Mike’s RSAC 2012 Wrap-up (securosis.com)
• Hacking breach made us stronger says RSA (go.theregister.com)
• Building/Bolting Security In/On – A Pox On the Audit Paradox! (rationalsurvivability.com)
• PSA: Paula Deen, Sausage Pancake Egg Sandwiches & Security… (rationalsurvivability.com)