Rational Survivability

I usually don’t spend much time when I write a blog, but this was ridiculously difficult to write.

I’m neither a neuroscientist or a computer scientist. I’ve dabbled in AI and self-organizing maps, but I can barely do fractions, so every sentence of this blog had me doubting writing it. It’s probably shit, but I enjoyed thinking about it.

The further I tried to simplify my thoughts, the less cogent they became and what spooled outward onto my screen resembled more porridge than prose.

That said, I often feel stymied while writing. When someone else has crystallized thoughts to which adding commentary seems panderous, redundant, or potentially intellectually fraudulent, it feels like there’s no possible way that my thoughts spilling out are original, credible, or meaningful.

This is especially the case as when brilliant people have written brilliant things on the topic.

“On the shoulders of giants” and all that…

Skynet, The Matrix, The Singularity, The Borg…all of these examples popped into my head as I wrote, destroying my almost sensical paragraphs with clumbsy analogs that had me longing to reduce my commentary to nothing more than basic Twitter and Facebook-like primitives: “< +1” or “Like.” It was all just a big pile of fail.

The funny thing is, that’s actually where this story begins and why its genesis was so intriguing.

Alex Williams wrote an article titled “How Machines Will Use Social Networks To Gain Identity, Develop Relationships And Make Friends.”

He offered up a couple of interesting examples from some conceptual “demos” from last week’s VMworld.  I re-read the article and found that the topic was profound, relevant and timely.

At its core, Alex challenges us to reimagine how “machines” — really combinations of infrastructure and applications that process information — might (self) identify, communicate, interoperate, organize and function as part of a collective construct, using a codified language that mimics the channels that we humans are today using in social patterns and grafs that define our relationships online.

The article wobbled a bit with the implication that machines might “feel,” but stripping relevant actions or qualitative measures such as “like” or “dislike” down to their core, it’s not hard to imagine how machines might evaluate or re-evaluate relationships, behavior and (re)actions based on established primitives such as “good,” “bad,” “available” or “malfuctioned.”

I know that’s how my wife generally thinks of me.

Frankly, it’s a simple concept. Even for humans. As an intelligently-complex species, humans define even heady things like emotional responses as a function of two fundamental neurotransmitters — chemical messengers — the biogenic amines serotonin and dopamine. The levels of these neurotransmitters are normally quite reasonably regulated but can be heightened or depressed based on the presence of and interaction with other chemical compounds. These neurochemical interactions may yield behavioral or even systemic immune system responses that manifest themselves in a variety of ways; from happiness to disease.

One might imagine that machines might likewise interact and form behavioral responses to, and thus relationships with, other groups of machines in either like-minded or opposing “clusters” using a distilled version of the very “activity streams” that humans feed into and out of using social media, defined by the dynamic, organic and chaotic social graph that ties them.

[I just noticed that my friend and prior colleague Mat Matthews from Plexxi wrote a blog on “affinity” and described this as Socially Defined Networks. Brilliant. ]

I’m sure that in some way, they already do. But again, I’m hung up on the fact that my NEST thermostat may actually be out to kill me…and tweet about it at an ecologically sound point in time when electricity costs are optimal.

The notion that machines will process these activity streams like humans do and act on them is really a natural extension of how today’s application architectures and infrastructure designs which utilize message buses and APIs to intercommunicate. It’s a bit of a re-hash of many topics and the autonomic, self-learning, HAL-9000 batshit crazy compute concepts we’ve all heard of before.

On Twitter, reacting to what he sensed as “sensationalism,” Thomas Lukasik (@sparkenstein) summarized my assessment of this concept (and thus rendering all these words even more useless) thusly:

“…my immediate response was that a “social network” is an ideal model 2 take advantage of N autonomous systems.”

My response: +1 (see what I did there? 😉

But what differentiates between the human social graph and the non-kinetic “cyber” graph is the capacity, desire and operational modality that describes how, when and why events are processed (or not.) That and crazy ex-girlfriends, pictures of dinner and politicial commentary.

I further addressed Thomas’ complaint that we’d seen this before by positing that “how humans are changing the way we interact will ultimately define how the machines we design will, too.”

To wit, machines don’t necessarily have the complexity, variety, velocity and volume of unrelated stimuli and distractions that humans do. We have more senses and we have fuzzy responses to binary responses. They are simpler, more discrete “creatures” and as their taskmasters, we enjoy a highly leveraged, somewhat predictable and reasonably consistent way in which they process and respond to events.

Usually until something kinetic or previously undefined occurs. Then, the dependency on automation and the ability for the discrete and systemic elements to “learn,” adapt, interact and leverage previously unrelated relationships with other nodes becomes important.  I wrote about that here: Unsafe At Any Speed: The Darkside Of Automation

What’s really relevant here, however,  is that the “social graph” approach — the relationship between entities and the policies established to govern them — can help close that gap.  Autonomous is cool.  Being part of an “autonomous collective” is cooler. As evidence, I offer up that scene with the peasants in Monty Python’s “Quest for the Holy Grail.”

In fact, if one were to look at computer networks, we’ve seen the evolution from centralized to distributed and now hybrid models of how the messages and state between entities are communicated and controlled.

Now, take a deep breath because I’m about to add yet another bit of “sensationalism” that Thomas will probably choke on…

The notion of separating the control, data and management planes that exist in the form of protocols and communication architectures are bubbling to the surface already in the highly-hyped area of software defined networking (SDN.)

I’m going to leave the bulk of my SDN example for another post, but bear with me for just a minute.  (Actually, this is where the blog descends into really crappily thought out rambling.)

If we have the capability to allow the applications and infrastructure — they’re both critical components of “the machine” — to communicate in an automated manner while contextualizing the notion that an event or message might indicate a need for state change, service delivery differences, or even something such as locality, and share this information with those who have a pre-defined relationship with a need-to-know, much goodness may occur.

Think: security.

This starts to bring back into focus the notion that like a human immune system, the ability to identify, localize and respond, signalling to the collective the disposition of the event and what may be needed to deal with it.

The implications are profound because as the systems of “machines” become increasingly more networked, adaptive and complex, they become more like living organisms and these collective “hives” will behave less like binary constructs, and much more like fuzzy communities of animals such as ants or bees.

If we bring this back into the teeniest bit more relevant focus — let’s say virtualized data centers or even (gasp!) Cloud, I think that collision between “social” and “networking” really can take on a broader meaning, especially within the context of how systems intercommunicate and interact with one another.

As an example, the orchestration, provisioning, automation and policy engines we’re deploying today are primitive. The fact that applications and infrastructure are viewed as discrete and not as a system further complicates the problem space because the paths, events, messages and actions are incomprehensible to each of these discrete layers.  This is why we can’t have nice things, America.

What’s coming, however, are really interesting collisions of relevant technology combined with fantastic applications of defining and leveraging the ways in which these complex systems of machines can become much more useful, interactive, communicative and “social.”

I think that’s what Alex was getting at when he wrote:

…points to an inevitable future. The machines will have a voice. They will communicate in increasingly human-like ways. In the near term, the advancements in the use of social technologies will provide contextual ways to manage data centers. Activity streams serve as the language that people understand. They help translate the interactions between machines so problems can be diagnosed faster.

By treating machines as individuals we can better provide visualizations to orchestrate complex provisioning and management tasks. That is inevitable in a world which requires more simple ways to orchestrate the increasingly dynamic nature for the ways we humans live and work with the machines among us.

Johnny Five is Alive.

Like.

Related articles

• How Machines Will Use Social Networks To Gain Identity, Develop Relationships And Make Friends
• Graphs are Everywhere: Solving the Complexities of Social Connections
• Prismatic Architecture – Using Machine Learning on Social Networks to Figure Out What You Should Read on the Web
• Understanding Open Graph Protocol

A “Potemkin village” is a Russian expression derived from folklore from the 1700’s.  The story goes something like this: Grigory Potemkin, a military leader and  statesman, erected attractive but completely fake settlements constructed only of facades to impress Catherine the Great (empress of Russia) during a state visit in order to gain favor and otherwise hype the value of recently subjugated territories.

I’ll get to that (and probably irate comments from actual Russians who will chide me for my hatchet job on their culture…)

Innovation over the last decade in technology in general has brought fundamental shifts in the way in which we work, live, and play. In the last 4 years, the manner in which technology products and services that enabled by this “digital supply chain,” and the manner in which they are designed, built and brought to market have also pivoted.

Virtualization and Cloud computing — the technologies and operational models — have contributed greatly to this.

Interestingly enough, the faster technology evolves, the more lethargic, fragile and fractured security seems to be.

This can be explained in a few ways.

First, the trust models, architecture and operational models surrounding how we’ve “done” security simply are not designed to absorb this much disruption so quickly.  The fact that we’ve relied on physical segregation, static policies that combine locality and service definition, mobility and the (now) highly dynamic application deployment options means that we’re simply disconnected.

Secondly, fragmentation and specialization within security means that we have no cohesive, integrated or consistent approach in terms of how we define or instantiate “security,” and so customers are left to integrate disparate solutions at multiple layers (think physical and/or virtual firewalls, IDP, DLP, WAF, AppSec, etc.)  What services and “hooks” the operating systems, networks and provisioning/orchestration layers offers largely dictates what we can do using the skills and “best practices” we already have.

Lastly, the (un)natural market consolidation behavior wherein aspiring technology startups are acquired and absorbed into larger behemoth organizations means that innovation cycles in security quickly become victims of stunted periodicity, reduced focus on solving specific problems, cultural subduction and artificially constrained scope based on P&L models which are detached from reality, customers and out of step with trends that end up driving more disruption.

I’ve talked about this process as part of the “Security Hamster Sine Wave of Pain.”  It’s not a malicious or evil plan on behalf of vendors to conspire to not solve your problems, it’s an artifact of the way in which the market functions — and is allowed to function.

What this yields is that when new threat models, evolving vulnerabilities and advanced adversarial skill sets are paired with massively disruptive approaches and technology “conquests,” the security industry  basically erects facades of solutions, obscuring the fact that in many cases, there’s not only a lacking foundation for the house of cards we’ve built, but interestingly there’s not much more to it than that.

Again, this isn’t a plan masterminded by a consortium of industry “Dr. Evils.”  Actually, it’s quite simple: It’s inertial…if you keep buying it, they’ll keep making it.

We are suffering then from the security equivalent of the Potemkin Village syndrome; our efforts are largely built to impress people who are mesmerized by pretty facades but don’t take the time to recognize that there’s really nothing there.  Those building it, while complicit, find it quite hard to change.

Until the revolution comes.

To wit, we have hardworking members of the proletariat, toiling away behind the scenes struggling to add substance and drive change in the way in which we do what we do.

Adding to this is the good news that those two aforementioned “movements” — virtualization and cloud computing — are exposing the facades for what they are and we’re now busy shining the light on unstable foundations, knocking over walls and starting to build platforms that are fundamentally better suited to support security capabilities rather than simply “patching holes.”

Most virtualization and IaaS cloud platforms are still woefully lacking the native capabilities or interfaces to build security in, but that’s the beauty of platforms (as a service,) as you can encourage more “universally” the focus on the things that matter most: building resilient and survivable systems, deploying secure applications, and identifying and protecting information across its lifecycle.

Realistically this is a long view and it is going to take a few more cycles on the Hamster Wheel to drive true results.  It’s frankly less about technology and rather largely a generational concern with the current ruling party who governs operational security awaiting deposition, retirement or beheading.

I’m looking forward to more disruption, innovation and reconstruction.  Let’s fix the foundation and deal with hanging pictures later.  Redecorating security is for the birds…or dead Russian royalty.

/Hoff

Related articles

• Is What We Need…An OpSec K/T Boundary Extinction-Level Event?
• Back To The Future: Network Segmentation & More Moaning About Zoning
• Elemental: Leveraging Virtualization Technology For More Resilient & Survivable Systems

Allow me to immediately state that I am, in no way, attempting to blame or shame the victim in my editorial below.

However, the recent rash of commentary from security wonks on Twitter and blogs regarding who is to “blame” in Mat Honan’s unfortunate experience leaves me confused and misses an important point.

Firstly, the title of the oft-referenced article documenting the series of events is at the root of my discontent:

“How Apple and Amazon Security Flaws Led to My Epic Hacking“

As I tweeted, my assessment and suggestion for a title would be:

How my poor behavior led to my epic hacking & flawed trust models & bad luck w/Apple and Amazon assisted

…especially when coupled with what is clearly an admission by Mr. Honan, that he is, fundamentally, responsible for enabling the chained series of events that took place:

In the space of one hour, my entire digital life was destroyed. First my Google account was taken over, then deleted. Next my Twitter account was compromised, and used as a platform to broadcast racist and homophobic messages. And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook.

In many ways, this was all my fault. My accounts were daisy-chained together. Getting into Amazon let my hackers get into my Apple ID account, which helped them get into Gmail, which gave them access to Twitter. Had I used two-factor authentication for my Google account, it’s possible that none of this would have happened, because their ultimate goal was always to take over my Twitter account and wreak havoc. Lulz.

Had I been regularly backing up the data on my MacBook, I wouldn’t have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location.

Those security lapses are my fault, and I deeply, deeply regret them.

The important highlighted snippets above are obscured by the salacious title and the bulk of the article which focuses on how services — which he enabled and relied upon — however flawed certain components of that trust and process may have been, are *really* at the center of the debate here.  Or ought to be.

There’s clearly a bit of emotional transference occurring.  It’s easier to associate causality with a faceless big corporate machine rather than swing the light toward the victim, even if he, himself, self-identifies.

Before you think I’m madly defending and/or suggesting that there weren’t breakdowns with any of the vendors — especially Apple — let me assure you I am not.  There are many things that can and should be addressed here, but leaving out the human element, the root of it all here, is dangerous.

I am concerned that as a community there is often an aire of suggestion that consumers are incapable and inculpable with respect to understanding the risks associated with the clicky-clicky-connect syndrome that all of these interconnected services brings.

People give third party applications and services unfettered access to services like Twitter and Facebook every day — even when messages surrounding the potential incursion of privacy and security are clearly stated.

When something does fail — and it does and always will — we vilify the suppliers (sometimes rightfully so for poor practices) but we never really look at what we need to do to prevent having to see this again: “Those security lapses are my fault, and I deeply, deeply regret them.”

The more interconnected things become, the more dependent upon flawed trust models and the expectations that users aren’t responsible we shall be.

This is the point I made in my presentations: Cloudifornication and Cloudinomicon.

There’s a lot of interesting discussion regarding the effectiveness of security awareness training.  Dave Aitel started a lively one here: “Why you shouldn’t train employees for security awareness”

It’s unfortunate the the only real way people learn is through misfortune, and any way you look at it, that’s the thing that drives awareness.

There are many lessons we can learn from Mr. Honan’s unfortunate experience…I urge you to consider less focusing blame on one link in the chain and instead guide the people you can influence to reconsider decisions of convenience over the potential tradeoffs they incur.

/Hoff

P.S. For you youngsters who don’t get the Soylent Green reference, see here.  Better yet, watch it. It’s awesome. Charlton Heston, FTW.

P.P.S. (Check out the sentiment of all the articles below)

Related articles

• Black cloud looms over Apple online service after high-profile hack
• Security lapses at Apple and Amazon lead to an epic hack. This could be you.
• Apple Responds To Journalist Victim of “Epic” Apple ID Hack
• iCloud attack began with Amazon hack
• Hack reveals Apple, Amazon’s gaping security flaws
• Getting epically hacked sounds lulzy till it happens to you

English: Defensive and offensive lines in American football (Photo credit: Wikipedia)

One cannot swing a dead cat without bumping into at least one expose in the mainstream media regarding how various nation states are engaged in what is described as “Cyberwar.”

The obligatory shots of darkened rooms filled with pimply-faced spooky characters basking in the green glow of command line sessions furiously typing are dosed with trademark interstitial fade-ins featuring the masks of Anonymous set amongst a backdrop of shots of smoky Syrian streets during the uprising,  power grids and nuclear power plants in lockdown replete with alarms and flashing lights accompanied by plunging stock-ticker animations laid over the trademark icons of financial trading floors.

Terms like Stuxnet, Zeus, and Flame have emerged from the obscure .DAT files of AV research labs and now occupy a prominent spot in the lexicon of popular culture…right along side the word “Hacker,” which now almost certainly brings with it only the negative connotation it has been (re)designed to impart.

In all of this “Cyberwar” we hear that the U.S. defense complex is woefully unprepared to deal with the sophistication, volume and severity of the attacks we are under on a daily basis.  Further, statistics from the Private Sector suggest that adversaries are becoming more aggressive, motivated, innovative, advanced,  and successful in their ability to attack what is basically described as basically undefended — nee’ undefendable — assets.

In all of this talk of “Cyberwar,” we were led to believe that the U.S. Government — despite hostile acts of “cyberaggression” from “enemies” foreign and domestic — never engaged in pre-emptive acts of Cyberwar.  We were led to believe that despite escalating cases of documented incursions across our critical infrastructure (Aurora, Titan Rain, etc.,) that our response was reactionary, limited in scope and reach and almost purely detective/forensic in nature.

It’s pretty clear that was a farce.

However, what’s interesting — besides the amazing geopolitical, cultural, socio-economic, sovereign,  financial and diplomatic issues that war of any sort brings — including “cyberwar” — is that even in the Private Sector, we’re still led to believe that we’re both unable, unwilling or forbidden to do anything but passively respond to attack.

There are some very good reasons for that argument, and some which need further debate.

Advanced adversaries are often innovative and unconstrained in their attack methodologies yet defenders remain firmly rooted in the classical OODA-fueled loops of the past where the A, “act,” generally includes some convoluted mixture of detection, incident response and cleanup…which is often followed up with a second dose when the next attack occurs.

As such, “Defenders” need better definitions of what “defense” means and how a silent discard from a firewall, a TCP RST from an IPS or a blip from Bro is simply not enough.  What I’m talking about here is what defensive linemen look to do when squared up across from their offensive linemen opponents — not to just hold the line to prevent further down-field penetration, but to sack the quarterback or better yet, cause a fumble or error and intercept a pass to culminate in running one in for points to their advantage.

That’s a big difference between holding till fourth down and hoping the offense can manage to not suffer the same fate from the opposition.

That implies there’s a difference between “winning” and “not losing,” with arbitrary values of the latter.

Put simply, it means we should employ methods that make it more and more difficult, costly, timely and non-automated for the attacker to carry out his/her mission…[more] active defense.

I’ve written about this before in 2009 “Incomplete Thought: Offensive Computing – The Empire Strikes Back” wherein I asked people’s opinion on both their response to and definition of “offensive security.”  This was a poor term…so I was delighted when I found my buddy Rich Mogull had taken the time to clarify vocabulary around this issue in his blog titled: “Thoughts on Active Defense, Intrusion Deception, and Counterstrikes.”

Rich wrote:

…Here are some possible definitions we can work with:

• Active defense: Altering your environment and system responses dynamically based on the activity of potential attackers, to both frustrate attacks and more definitively identify actual attacks. Try to tie up the attacker and gain more information on them without engaging in offensive attacks yourself. A rudimentary example is throwing up an extra verification page when someone tries to leave potential blog spam, all the way up to tools like Mykonos that deliberately screw with attackers to waste their time and reduce potential false positives.
• Intrusion deception: Pollute your environment with false information designed to frustrate attackers. You can also instrument these systems/datum to identify attacks. DataSoft Nova is an example of this. Active defense engages with attackers, while intrusion deception can also be more passive.
• Honeypots & tripwires: Purely passive (and static) tools with false information designed to entice and identify an attacker.
• Counterstrike: Attack the attacker by engaging in offensive activity that extends beyond your perimeter.

These aren’t exclusive – Mykonos also uses intrusion deception, while Nova can also use active defense. The core idea is to leave things for attackers to touch, and instrument them so you can identify the intruders. Except for counterattacks, which move outside your perimeter and are legally risky.

I think that we’re seeing the re-emergence of technology that wasn’t ready for primetime now become more prominent in consideration when folks refresh their toolchests looking for answers to problems that “passive response” offers.  It’s important to understand that tools like these — in isolation — won’t solve many complex attacks, nor are they a silver bullet, but understanding that we’re not limited to cleanup is important.

The language of “active defense,” like Rich’s above, is being spoken more and more.

Traditional networking and security companies such as Juniper* are acquiring upstarts like Mykonos Software in this space.  Mykonos’ mission is to “…change the economics of hacking…by making the attack surface variable and inserting deceptive detection points into the web application…mak[ing] hacking a website more time consuming, tedious and costly to an attacker. Because the web application is no longer passive, it also makes attacks more difficult.”

VC’s like Kleiner Perkins are funding companies whose operating premise is a more active “response” such as the in-stealth company “Shape Security” that expects to “…change the web security paradigm by shifting costs from defenders to hackers.”

Or, as Rich defined above, the notion of “counterstrike” outside one’s “perimeter” is beginning to garner open discussion now that we’ve seen what’s possible in the wild.

In fact, check out the abstract at Defcon 20 from Shawn Henry of newly-unstealthed company “Crowdstrike,” titled “Changing the Security Paradigm: Taking Back Your Network and Bringing Pain to the Adversary:”

The threat to our networks is increasing at an unprecedented rate. The hostile environment we operate in has rendered traditional security strategies obsolete. Adversary advances require changes in the way we operate, and “offense” changes the game.

Shawn Henry Prior to joining CrowdStrike, Henry was with the FBI for 24 years, most recently as Executive Assistant Director, where he was responsible for all FBI criminal investigations, cyber investigations, and international operations worldwide.

If you look at Mr. Henry’s credentials, it’s clear where the motivation and customer base are likely to flow.

Without turning this little highlight into a major opus — because when discussing this topic it’s quite easy to do so given the definition and implications of “active defense,”– I hope this has scratched an itch and you’ll spend more time investigating this fascinating topic.

I’m convinced we will see more and more as the cybersword rattling continues.

Have you investigated technology solutions that offer more “active defense?”

/Hoff

* Full disclosure: I work for Juniper Networks who recently acquired Mykonos Software mentioned above.  I hold a position in, and enjoy a salary from, Juniper Networks, Inc. 😉

Related articles

• Thoughts on Active Defense, Intrusion Deception, and Counter-strikes(securosis.com)

  ---

• No National ‘Stand Your Cyberground’ Law Please(infosec3t.com)

  ---

• Juniper Networks enhances its Mykonos Web Security software(net-security.org)

  ---

• Hacked companies fight back with controversial steps(reuters.com)

  ---

• President complains about his exposed cyberwar(dickdestiny.com)

  ---

• Premature Counter Offensive Actions Could Yield Painful Results(blogs.rsa.com)

  ---

As I previously wrote, I attended the GigaOm Structure Conference and was fortunate enough to participate in a chat with Stacey Higginbotham (GigaOm) and Simon Crosby (Bromium.)

In the beginning of our session, after Simon’s “unveiling” of Bromium’s approach to solving some tough security challenges, we engaged in some dialog about those same security challenges [for context] and many broader security topics in general.  Stacey led off by rhetorically asking me if I was an advisor to Bromium.  I answered in the affirmative with one word, “yes.”

To add more color, what “yes” meant was that I have advised leadership and employees of Bromium as to their approach, technology and productization and have access to their “technology preview” (read: beta) program.

What I didn’t  clarify is that like every other opportunity wherein I “advise” individuals, boards, companies or investors (institutional or otherwise,) I do not receive compensation for such activities.  No stock, bonds, gifts, cash, etc. The only thing that might qualify as compensation is when I have to travel to a remote location that I can’t expense myself or my employer can’t/won’t cover.  Every one in a while, I get a meal out of these activities so we can do a brain-dump outside of normal working hours so my employer is not impacted.

Those things may, in some people’s eyes, still seem like “compensation.”  I think that’s fair enough.  I’m also required to disclose any position I undertake with my employer to avoid conflict of any sort.

This is interesting to me because I’ve never really thought about disclosing any further my “advisory” roles outside of this process because I never put myself in the position wherein I feel either I have a vested interest (financially) in the company’s outcome.  The reason I “advise” is that it allows me early stage access to very interesting topics that I (cautiously) comment on — both publicly or privately where appropriate — and everyone involved wins.

What motivated this was a private DM exchange between someone (an analyst who shall remain nameless but to whom I am thankful) who attended Structure and was kind and honest enough to tell me what he thought.  Specifically, he suggested I had “crossed the line” in my public “endorsement” of Bromium.

Check out the thread below.  I found it fascinating.  To me, this seemed to be one part poor communication/disclosure on my part regarding what being an “advisor” entailed and one part complaint that perhaps I was messing up the business model of those who advise for free.

There was one additional point made that to the investment world, there’s a distinction between investing, advising and mentoring wherein “mentoring” was the only category that implied there was no financial compensation.  I’ve never really thought about making the distinction because again I’ve never asked for compensation…so I guess I’ve been a “mentor,” but I’d feel awkward calling myself that.

At any rate, I learned something from the exchange.  Maybe you will, too.

/Hoff

(Photo credit: IvanWalsh.com)

I’m about to get in a metal tube and spend 14 hours in the Clouds.  I figured I’d get something off my chest while I sit outside in the sun listening to some Jimmy Buffett.

[Network] overlays.  They bug me.  Let me tell you why.

The Enterprise, when considering “moving to the Cloud” generally takes one of two approaches depending upon culture, leadership, business goals, maturity and sophistication:

1. Go whole-hog with an all-in Cloud strategy. 
   Put an expiration date on maintaining/investing in legacy apps/infrastructure and instead build an organizational structure, technology approach, culture, and operational model that is designed around building applications that are optimized for “cloud” — and that means SaaS, PaaS, and IaaS across public, private and hybrid models with a focus on how application delivery and information (including protecting) is very different than legacy deployments, or…
2. Adopt a hedging strategy to get to Cloud…someday.
   This usually means opportunistically picking low risk, low impact, low-hanging fruit that can be tip-toed toward and scraping together the existing “rogue” projects already underway, sprinkling in some BYOD, pointing to a virtualized datacenter and calling a 3 day provisioning window with change control as “on-demand,” and “Cloud.”  Oh, and then deploying gateways, VPNs, data encryption and network overlays as an attempt to plug holes by paving over them, and calling that “Cloud,” also.

See that last bit?

This is where so-called “software defined networking (SDN),” the myriad of models that utilize “virtualization” and all sorts of new protocols and service delivery mechanisms are being conflated into the “will it blend” menagerie called “Cloud.”  It’s an “eyes wide shut” approach.

Now, before you think I’m being dismissive of “virtualization” or SDN, I’m not.  I believe. Wholesale. But within the context of option #2 above, it’s largely a waste of time, money, and effort.  It’s putting lipstick on a pig.

You either chirp or get off the twig.

Picking door #2 is where the Enterprise looks at shiny new things based on an article in the WSJ, Wired or via peer group golf outing and says “I bet if we added yet another layer of abstraction atop the piles of already rapidly abstracting piles of shite we already have, we would be more agile, nimble, efficient and secure.”  We would be “cloud” enabled.

[To a legacy-minded Enterprise,] Cloud is the revenge of VPN and PKI…

The problem is that just like the folks in Maine will advise: “You can’t get there from here.”  I mean, you can, but the notion that you’ll actually pull it off by stacking turtles, applying band-aids and squishing the tyranny of VLANs by surrounding them in layer 3 network overlays and calling this the next greatest thing since sliced bread is, well, bollocks.

Look, I think SDN, protocols like Openflow and VXLAN/NVGRE, etc. are swell.  I think the separation of control and data planes and the notion that I can programmatically operate my network is awesome.  I think companies like Nicira and Bigswitch are doing really interesting things.  I think that Cloudstack, Openstack and VMWare present real opportunity to make things “better.”

Hey, look, we’re just like Google and Amazon Web Services Now!

But to an Enterprise without a real plan as to what “Cloud” really means to their business, these are largely overlays within the context of #2.  Within the context of #1, they’re simply mom and apple pie and are, for the most part, invisible.  That’s not where the focus actually is.

That said, for a transitional Enterprise, these things give them pause, but should be looked upon as breadcrumbs that indicate a journey, not the destination.  They’re a crutch and another band-aid to solve legacy problems.  They’re really a means to an end.

These “innovations” *are* a step in the right direction.  They will let us do great things. They will let a whole new generation of operational models and a revitalized ecosystem flourish AND it will encourage folks to think differently.  But about what?  And to solve what problem(s)?

If you simply expect to layer them on your legacy infrastructure, operational models and people and call it “Cloud,” you’re being disingenuous.

Ultimately, to abuse an analogy, network overlays are a layover on the itinerary of our journey to the Cloud, but not where we should ultimately land. I see too many companies focusing on the transition…and by the time they get there, the target will have moved.  Again.  Just like it always does.

They’re hot now because they reflect something we should have done a long time ago, but like hypervisors, one day [soon] network overlays will become just a feature and not a focus.

/Hoff

Related articles

• Incomplete Thought: Will the Public Cloud Create a Generation Of Network Stupid?(rationalsurvivability.com)

  ---

• Building/Bolting Security In/On – A Pox On the Audit Paradox! (rationalsurvivability.com)
• AwkwardCloud: Here’s Hopin’ For Open (rationalsurvivability.com)

My buddy Bill Brenner (@billbrenner70) blogged a question that stemmed from a “discussion” I seem to have initiated yesterday: “Do People In Security Blog Too Much?”

He was kind enough to accommodate a clarification from me in which I reiterated that my chief complaint regarding excessive self-promotion by individuals  was “not about volume, but variety.”

To be clear, RT’ing a link (however modified) that is clearly designed to self-promote onesself is, in my opinion, bordering on SPAM-like behavior when one does it 10+ times in a 24 hour period.

I don’t mind a lot of tweets.  I mind a lot of the same tweets.

…The same way people get annoyed with folks who live tweet conferences, I suppose.

Now, people have the right to tweet whatever they like, as often as they like, but the reason I brought this up was because I was truly interested in whether or not the individual in question understood the impact/annoyance it caused.

Based on his reply, the “data he had to suggest ‘increased engagement,’ and what was clearly a strategy behind this activity, it became apparent he didn’t.

So I did what anyone in my position has the option to do: I unfollowed.  This was followed by an additional comment from the author that only “…~0.1% of followers had a negative response” to his RT’ing [approximately 5/4200 people.]

I found that odd, since I had at least 10 DM’s in my mailbox from followers who reacted to my tweets surrounding this issue.

5 or so others then piped up suggesting they were also annoyed but, like me, had not said anything.

As I mentioned, I wasn’t looking for anything like an apology — it’s not my place to, nor am I arrogant enough to suggest I’m owed one — but I did want him to understand that there were ramifications that either he was unaware of or simply ignoring.  Again, his choice.

I probably *do* tweet too much for many people’s likes — and they unfollow accordingly.  However, I operate under the “code” that I try very hard to not RT anything self-promotional more than TWICE in a 24 hour period.  I figure that with timezone deltas, but with RSS feeds and other RT’s from interested parties, that’s sufficient.

Am I potentially missing people?  Sure.  But the way I look at it is that if it’s interesting enough, people will find it.

I’m not in the “business” of “SEO for Twitter” (h/t to @SecureTom for the phrase,) but that’s a personal choice.

I will suggest, however, that people are smarter than many give them credit for — you can get cute and change the preamble, but if you deluge their timeline with self-promotion, expect them to one day get grumpy enough to find the unfollow button…and use it.

/Hoff

Here they are…*some* of my favorite Tweets O’ the Week that I curated:

• Unless you like fish, stop chasing red herrings.
• The hypervisor is/should be the least of your security concerns in a virtualized environment. The ops & mgmt layer should be
• The next 1 of you (us) who starts whining about how broken our industry is without doing anything about it gets posted to the hamster wall
• This is the new norm I call anti-FUD FUD: security vendors shitting where they eat in an (em)pathetic attempt to gain cred. How ’bout fixin?
• Congrats on $60MM funding @appirio. It’s great u’ll be able to afford to create even more BS marketing contests you rig the outcome to ;p
• Protip: The state of the Security Industry always looks like shit in the middle of a “breaker” hacker con.  By design. You’re welcome.
• More negativity, navel gazing & security apocalypse hype. Funny how “experts” doing the sky-is-falling chicken dance never propose solutions
• Awkward moment today: someone presenting me slides re: Cloud Security that I built on an initiative I created and a group I lead. o_O
• Oh! Right! Cloud security, visibility & transparency. Why didn’t I think of that?!
• North by Northwest is basically the Hitchcock version of Anonymous, Wikileaks…with biplanes and better acting.
• I will soon utilize HTTPS/SSL to encrypt all my tweets. Those of you who are not Beaker Certified will be unable to decipher my madness
• Out of complete ignorance: is SXSW like Burning Man for nerds who only discuss things that are battery operated?
• What a bunch of chicken shits. 20 DM’s later and 18 of you vote @MikD as the Ryan Seacrest of Infosec. Like that’s a bad thing?
• My twitter follower count goal is 90210 – that way I can claim I am the Tiffany Amber Theisen of Twitter. It’s the little things…
• Single best way to get uninvited back to weekly meetings is introduce the fact that the host’s model construct for an argument is flawed.
• Oh $gawd. What a bunch of cockblocking going on with respect to $openwashing & who started what. Sigh. #getonwithitalready
• I just sent the most awesome f’ing internal email ever.  If there was EVER a reason for REPLY-ALL, *this* would be it. GRAB YOUR RED STAPLER

Did I miss any? 😉

Update 030712: I’m going to follow this post up with yet another post mortem that includes lessons learned and more details as I can supply them.  I will point out two things:

1. It’s pretty clear that the secondary/tertiary stages of this infestation which led to multiple alerts from my readers is related to the massive WordPress attack you can read about here.  It’s important to note, however, that the first incident (which was chalked up impoperly to a false positive) and a second started with similar symptoms back in late July.  I simply didn’t have the data to correlate.  They were different variants.
2. The support from vendors and the security community has been outstanding.  People with no vested interest in the health of my personal blog have gone out of their way to help, even my hosting provider, Dreamhost (although we got off on a rather rocky footing 😉

I do owe both FireEye (who spotted the original attack) and Dreamhost additional data which I will attempt to retrieve.  I also owe Rich Mogull an apology/explanation regarding why I didn’t immediately take the blog down, risking further infection — I legitimately thought we’d fixed it, but because of the stealth of the malware, I was wrong.  Once I realized I couldn’t contain/isolate it, I did take it down…and then wiped the entire blog/database.

At any rate, thanks for bearing with me though this.  It’s been invaluable to me and I hope you found some value in all of this.

It certainly was interesting and gave me some unique insight into the psychology, behavior, biases and opinions of the community/industry that I didn’t fully appreciate prior.

—

This is an update that I originally included with the post describing the malicious infestation of malware on my WordPress site here.  I’ve split it out for clarity.

The last 12 hours or so have been fun. I’ve had many other folks join in and try to help isolate and eradicate the malware that plagued my WordPress install (read the original post below.)

I was able to determine that the Dreamhost password compromise in January (correlated against logs) was responsible for the (likely) automated injection of malicious PHP code into a plug-in directory that had poor permissions.  This code was BASE64 encoded. It was hard to find.

Further, as was alluded to in my earlier version of this post, the malware itself was adaptive and would only try (based on UA and originating IP) to drop it’s Windows-based trojan executable ONCE by way of a hidden iFrame. Hit it again and you’d never see it.

It was a variant of the Blackhole Exploit kit.

If you ran any up-to-date AV solution (as evidenced by the 6 different brands that people reported,) visiting my site immediately tripped an alert.  I run a Mac and up until today didn’t have such a tool installed. I clearly do now as a detective capability.  This was a silly thing NOT to do as it costs basically nothing to do so these days.

When I made a backup of the entire directory, my VPS hosting provider THEN decided to run a security scan on the directory (serendipity) and notified me via email that it found the malware in the directory 🙁 Thanks.  Great timing.  The funny thing was that all the activity last night and uploaded telemetry must have set something off in Google because only late last night — 30+ days later — did Google flag the site as potentially compromised.  Sigh.

At any rate, I ended up nuking my entire WordPress and mySQL installations and doing a fresh install. I’ve rid myself of almost every plug-in and gone back to a basic theme.  I’ve installed a couple of other detective and preventative tools on the site and will likely end up finally putting the site behind CloudFlare for an additional layer of protection.

Really, I should have done this stuff LONG ago…this was my personal failure.  I owe it to the kindness and attentiveness of those who alerted me to the fact that their AV sensors tripped.

The interesting note is that most of the security pros I know who run Macs and have visited my site in the last 30 days never knew I was infected.  If this were a Mac-targeted malware, perhaps they may have been infected.  The point is that while I’m glad it didn’t/couldn’t infect Mac users, I do care that I could have harmed users with other operating systems.

Further, the “ignorance is bliss” approach is personally alarming to me; without a tool which many security pros sleight as “useless,” I would never have know I was infected.

If anything, it should make you think…

(I originally pre-pended to this post a lengthy update based on my findings and incident response, but per a suggestion from @jeremiahg, I’ve created a separate post here for clarity)

Earlier today I wrote about the trending meme in the blogosphere/security bellybutton squad wherein the notion that security — or the perceived lacking thereof — is losing the “war.”

My response was that the expectations and methodology by which we measure success or failure is arbitrary and grossly inaccurate.  Furthermore, I suggest that the solutions we have at our disposal are geared toward solving short-term problems designed to generate revenue for vendors and solve point-specific problems based on prevailing threats and the appetite to combat them.

As a corollary, if you reduce this down to the basics, the tools we have at our disposal that we decry as useless often times work just fine…if you actually use them.

For most of us, we do what we can to provide appropriate layers of defense where possible but our adversaries are crafty and in many cases more skilled.  For some, this means our efforts are a lost cause but the reality is that often times good enough is good enough…until it isn’t.

Like it wasn’t today.

Let me paint you a picture.

A few days ago a Wired story titled “Is antivirus a waste of money?” hit the wires that quoted many (of my friends) as saying that security professionals don’t run antivirus.  There were discussions about efficacy, performance and usefulness. Many of the folks quoted in that article also run Macs.  There was some interesting banter on Twitter also.

If we rewind a few weeks, I was contacted by two people a few days apart, one running a FireEye network-based anti-malware solution and another running a mainstream host-based anti-virus solution.

Both of these people let me know that their solutions detected and blocked a Javascript-based redirection attempt from my blog which runs a self-hosted WordPress installation.

I pawed through my blog’s PHP code, turned off almost every plug-in, ran the exploit scanner…all the while unable to reproduce the behavior on my Mac or within a fresh Windows 7 VM.

The FireEye report ultimately was reported back as a false positive while the host-based AV solution couldn’t be reproduced, either.

Fast forward to today and after I wrote the blog “You know what’s dead? Security…” I had a huge number of click-throughs from my tweet.

The point of my blog was that security isn’t dead and we aren’t so grossly failing but rather suffering a death from a thousand cuts.  However, while we’ve got a ton of band-aids, it doesn’t make it any less painful.

Speaking of pain, almost immediately upon posting the tweet, I received reports from 5-6 people indicating their AV solutions detected an attempted malicious code execution, specifically a Javascript redirector.

This behavior was commensurate with the prior “sightings” and so with the help of @innismir and @chort0, I set about trying to reproduce the event.

@chort0 found that a hidden iFrame was redirecting to a site hosting in Belize (screen caps later) that ultimately linked to other sites in Russia and produced a delightful greeting which said “Gotcha!” after attempting to drop an executable.

Again, I was unable to duplicate and it seemed that once loaded, the iFrame and file dropper did not reappear.  @innismir didn’t get the iFrame but grabbed the dropped file.

This led to further investigation that it was likely this was an embedded compromise within the theme I was using.  @innismir found that the Sakura theme included “…woo-tumblog [which] uses a old version of TimThumb, which has a hole in it.”

I switched back to a basic built-in theme and turned off the remainder of the non-critical plug-ins.

Since I have no way of replicating the initial drop attempt, I can only hope that this exercise which involved some basic AV tools, some browser debug tools, some PCAP network traces and good ole investigation from three security wonks has paid off…

ONLY YOU CAN PREVENT MALWARE FIRES (so please let me know if you see an indication of an attempted malware infection.)

Now, back to the point at hand…I would never have noticed this (or more specifically others wouldn’t) had they not been running AV.

So while many look at these imperfect tools as a failure because they don’t detect/prevent all attacks, imagine how many more people I may have unwittingly infected accidentally.

Irony?  Perhaps, but what happened following the notification gives me more hope (in the combination of people, community and technology) than contempt for our gaps as an industry.

I plan to augment this post with more details and a conclusion about what I might have done differently once I have a moment to digest what we’ve done and try and confirm if it’s indeed repaired.  I hope it’s gone for good.

Thanks again to those of you who notified me of the anomalous behavior.

What’s scary is how many of you didn’t.

Is security “losing?”

Ask me in the morning…I’ll likely answer that from my perspective, no, but it’s one little battle at a time that matters.

/Hoff

Related articles

• WordPress Hacked, Security Steps (Astroarch)
• Security professionals DO use anti-virus (eset.com)
• Report: malware pushed by affiliate networks remains the primary growth factor of the cybercrime ecosystem (zdnet.com)
• A Blogger’s Worst Nightmare – Your site has been hacked by malware! (bpwebnews.com)
• You Know What’s Dead? Security… (rationalsurvivability.com)