Rational Survivability

While I think Nick Carr’s power generation utility analogy was a fantastic discussion catalyst for the usefulness of a utility model, it is abused to extremes and constrains what might ordinarily be more open-minded debate on the present and future of computing.

This is a debate that continues to rise every few days on Twitter and the Blogosphere, fueled mostly by what can only be described from either side of the argument as a mixture of ideology, dogma, passionate opinion, misunderstood perspective and a squinty-eyed mistrust of agendas.

It’s all a bit silly, really, as both Public and Private Cloud have their place; when, for how long and for whom is really at the heart of the issue.

The notion that the only way “true” benefits can be realized from Cloud Computing are from massively-scaled public utilities and that Private Clouds (your definition will likely differ) are simply a way of IT making excuses for the past while trying to hold on to the present, simply limits the conversation and causes friction rather than reduces it.  I believe that a hybrid model will prevail, as it always has.  There are many reasons for this. I’ve talked about them a lot.

This got me thinking about why and here’s my goofy thought for consideration of the “value” and “utility” of Private Cloud:

If the power utility “grid” represents Public Cloud, then perhaps batteries are a reasonable equivalent for Private Cloud.

I’m not going to explain this analogy in full yet, but wonder if it makes any sense to you.  I’d enjoy your thoughts on what you think I’m referring to. 😉

/Hoff

I think perhaps my choice of words were met with an unfortunate style of punctuation I was not expecting…

The Internet — once again kicking security’s ass, Karate Kid style, no less…

It seems I’m going to have to sharpen my mad skills, as the previous two meetings have led to similar results:

and…

There’s been a very interesting set of discussions lately regarding performance anomalies across Cloud infrastructure providers.  The most recent involves Amazon Web Services and RackSpace Cloud. Let’s focus on the former because it’s the one that has a good deal of analysis and data attached to it.

Reuven Cohen’s post (Oversubscribing the Cloud) summarizing many of these concerns speaks to the meme wherein he points to Alan Williamson’s initial complaints (Has Amazon EC2 become over subscribed?) followed by CloudKick’s very interesting experiments and data (Visual Evidence of Amazon EC2 network issues) and ultimately Rich Miller’s summary including a response from Amazon Web Services (Amazon: We Don’t Have Capacity Issues)

The thing that’s interesting to me in all of this is yet another example of people mixing metaphors, terminology and common operating methodologies as well as choosing to suspend disbelief and the reality distortion field associated with how service providers actually offer service versus marketing it.

Here’s the kicker: over subscription is not the same thing as over capacity. BY DESIGN, modern data/telecommuication (and Cloud) networks are built using an over-subscription model.

On the other hand, the sad truth is that we will have over capacity issues in cloud; it’s simply a sad intersection of the laws of physics and the delicate balance associated with cost control and service delivery.

Let me frame the following with an example: when you purchase an “unlimited data plan” from a telco or hosting company, you’ll notice normally that this does not have latency or throughput figures attached to it…same with Cloud.  You shouldn’t be surprised by this. If you are, you might want to rethink your approach to service level expectation.

Short and sweet:

1. There is no such thing as infinite scale.  There is no such thing as an “unlimited ____ plan.”* Even in Cloud. Every provider has limits, even if they’re massive. Adding the word Cloud simply squeezes the limit balloon from you to them and it’s a tougher problem to solve at scale. It doesn’t eliminate the issue, even with “elasticity.”
2. Allow me to repeat: over subscription is not the same thing as over capacity. BY DESIGN, modern data/telecommuication (and Cloud) networks are built using an over-subscription model.  I don’t need to explain why, I trust.
3. Capacity refers to the ability, within service level specifications, to meet the contracted needs of the customer and operate within acceptable thresholds. Depending upon how a provider measures that and communicates it to you, you may be horribly surprised if you chose the marketing over the engineering explanations of such.
4. Capacity is also not the same as latency, is not the same as throughput…
5. Over capacity means that the provider’s over-subscription modeling was flawed and suggests that the usage patterns overwhelmed the capacity threshold and they had no way of adding capacity in a manner which allows them to satisfy demand

Why is this important?  Because the “illusion” of infinite scale is just that.

The abstraction at the infrastructure layer of compute, network and storage — especially delivered in software — still relies on the underlying capacity of the pipes and bit-buckets that deliver them. It’s a never-ending see-saw movement of Metcalfe’s and Moore’s laws.

The discrete packaging of each virtualized CPU compute element sizing within an AWS or Rackspace is relatively easy to forecast and yields a reasonably helpful “fixed” capacity planning data point; it has a minima of zero and a maxima associated with the peak compute hours/vCPU clock rating of the instance.

The network piece and its relationship to the compute piece is where it gets interesting.  Your virtual interface ultimately is bundled together in aggregate with other tenants colocated on the same physical host and competes for a share of pipe (usually one or more single or trunked 1Gb/s or 10Gb/s Ethernet.) Network traffic in terms of measurement, capacity planning and usage must take into consideration the facts that it is both asymmetric, suffers from variability in bucket size, and is very, very bursty. There’s not generally a published service level associated with throughput in Cloud.

This complicates things when you consider that at this point scaling out in CPU is easier to do than scaling out in the network.  Add virtualization into the mix which drives big, flat, L2 networks as a design architecture layered with a control plane that is now (in the case of Cloud) mostly software driven, provisioned, orchestrated and implemented, and it’s no wonder that folks like Google, Amazon and Facebook are desparate for hugely dense, multi-terabit, wire speed L2 switching fabrics and could use 40 and 100Gb/s Ethernet today.

Check out this interesting article.

Oh, let’s not forget that there are also now providers who are deploying converged data/storage networking of said pipes with the likes of FCoE/DCE with all sorts of interesting ramifications on the above discussion.  If you thought it was tough to get your arms around before…

If you know much about Ethernet, congestion avoidance/recovery/control, QoS, etc. you know that it’s a complex beast. If service levels relating to network performance aren’t in your contract, you’re probably figuring out why right about now.

So, wrapping this up, I have to accept AWS’ statement that they “…do not have over-capacity issues,” because quite frankly there’s nothing to suggest otherwise.  That’s not to say there aren’t performance issues are related to something else (like software or hardware in the stack) but that’s not the same as being over capacity — and you’ll notice that they didn’t say they were not “over-subscribed” but rather they were not “over capacity.” 😉

/Hoff

*Just ask AT&T about their network and the iPhone. This *is* a case where their over-subscription planning failed in the face of capacity…and continues to.

It’s full of awesomesauce.

Here.

Cloud Light Presents…Real Men of Genius
{Real Men of Genius…}

Today we salute you, Mr. Dump-All-Your-Crap-In-the-Cloud Guy
{Mr. Dump-All-Your-Crap-In-the-Cloud Guy}

Some seek danger in cliff diving…others? Competitive eating…flamethrowing or ferret wrestling. But You? You put data in other people’s hands in the Cloud
{You’re asking for it}

Armed with a SAS-70 and a license to commit PCI, you live your life with a simple code: Finders keepers, losers weepers
{Finders Keepers}

Some people mock you, sure. But you paid $8.32 for your EC2 spot instances and well, you just can’t get that from Dreamhost
{who’s laughin’ now?}

So crack open a cloud instance, oh King of the Cloud…we’d give you our data, but you’ve probably already lost it
{Mr. Dump-All-Your-Crap-In-the-Cloud Guy}

Cloudheiser Bushed, Poughkipsie, New Jersey…

If you’re interested in the great discussion and presentations we had during the kickoff call for the A6 (Automated Audit, Assertion, Assessment, and Assurance API) Working Group, there are two options to listen/view the WebEx recording:

Topic: A6 API Working Group – Kickoff Call-20100108 1704
Create time: 1/8/10 10:07 am
File size: 33.23MB
Duration: 1 hour 1 minute
Description: Streaming recording link:
https://ciscosales.webex.com/ciscosales/ldr.php?AT=pb&SP=MC&rID=41631852rKey=178e8b04941e5672
Download recording link:
https://ciscosales.webex.com/ciscosales/lsr.php?AT=dw&SP=MC&rID=41631…

MAKE SURE YOU VIEW THE CHAT WINDOW << It contains some really excellent discussion points.

We had two great presentations from representatives from the OGF OCCI group and CSC’s Trusted Cloud Team.

I’ll be setting up regular calls shortly and a few people have reached out to me regarding helping form the core team to begin organizing the working group in earnest.

You can also follow along via the Google Group here.

/Hoff

In need of a cool logo for the group by the way… 😉

Back in October 2008, I wrote a post detailing efforts around the Xen community to create a standard security introspection API (Xen.Org Launches Community Project To Bring VM Introspection to Xen🙂

The Xen Introspection Project is a community effort within Xen.org to leverage the existing research presented above with other work not yet public to create a standard API specification and methodology for virtual machine introspection.

That blog was focused on introspection for virtualization proper but since many of the larger cloud providers utilize Xen virtualization as an underpinning of their service architecture and as an industry we’re suffering from a lack of visibility and deployable security capabilities, the relevance of VM and VMM introspection to cloud computing is quite relevant.

I thought I’d double around and see where we are.

It looks as though there’s been quite a bit of recent activity from the folks at Georgia Tech (XenAccess Project) and the University of Alaska at Fairbanks (Virtual Introspection for Xen) referenced in my previous blog.  The vCloud API proffered via the DMTF seems to also leverage (at least some of) the VMsafe API capabilities present in VMware‘s vSphere virtualization platform.

While details are, for obvious reasons sketchy, I am encouraged in speaking to representatives from a few cloud providers who are keenly interested in including these capabilities in their offerings.  Wouldn’t that be cool?

Adoption and inclusion of introspection capabilities will overcome some of the inherent security and visibility limitations we face in highly-virtualized multi-tenant environments due to networking constraints for integrating security functionality that I wrote about here.

I plan a follow-on blog in more detail once I finish some interviews.

/Hoff

Related articles by Zemanta

• Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… (rationalsurvivability.com)
• The Cloud & eHarmony’s 29 Dimensions Of Compatability… (rationalsurvivability.com)
• Incomplete Thought: Virtual Machines Are the Problem, Not the Solution… (rationalsurvivability.com)
• Variety & Darwinism In Solutions Is Innovation, In Standards It’s A War? (rationalsurvivability.com)
• Silent Lucidity: IaaS – Already A Dinosaur? The Evolution of PaaSasaurus Rex… (rationalsurvivability.com)
• Incomplete Thought: Storage In the Cloud: Winds From the ATMOS(fear) (rationalsurvivability.com)
• Redux: Patching the Cloud (rationalsurvivability.com)
• Cloud: Security Doesn’t Matter (Or, In Cloud, Nobody Can Hear You Scream) (rationalsurvivability.com)
• The Emotion of VMotion… (rationalsurvivability.com)
• Standards-Based Virtualization: Critical To The Future Of Cloud Computing (techcrunchit.com)
• VMware cloud initiative raises vendor lock-in concerns (infoworld.com)
• Where Are the Network Virtual Appliances? (gigaom.com)
• Is Virtualization Magic? (And Other Questions Your Manager May Ask) (readwriteweb.com)
• A Fluid Network is the Result of Collaboration Not Virtualization (devcentral.f5.com)
• The Road To Open Federated Clouds: Xen, VMware And More (cloudave.com)

I TRIPLE-DOG-DARE You!

There’s an awful lot of hyperbole being flung back and forth about the general state of security and Cloud-based services.

I’ve spent enough time highlighting both the practical and hypothetical (many of which actually have been realized) security issues created and exacerbated by Cloud up and down the stack, from IaaS to SaaS.

It seems, however, that there are a select few who ignore issues brought to light and seem to suggest that Cloud providers are at a state of maturity wherein they not only offer parity, but offer better security than the “average” IT shop.  What’s interesting is that while I agree that “Cloud Security is not insurmountable,” neither is non-Cloud security — but it’s sure as hell not progressed much in 40 years.

What’s missing is context.  What’s missing is the very risk assessment methodologies they reference in their tales of fancy.  What’s missing is that in the cases they suggest that security is not an obstacle to Cloud, there’s usually not much sensitive data or applications involved.

Ignore the U.S. CIO’s words of wisdom when he discusses the reality of security and moving to the Cloud. Ignore the CIO’s and CISO’s of the Fortune 500. Ignore everything in my Cloudifornication presentation and recent issues related to such. Ignore pragmatism.

Take my challenge instead…Here’s my dare:

1. I’ll pay for an AWS EC2 instance for a month
2. You choose the OS and LAMP stack components you’ll deploy in this AMI
3. You harden it however you see fit, but ensure the web server can be reached via port 80 from the Internet*
4. You put a .txt file somewhere on a readable filesystem (mounted) or create a row in a DB accessible via the web server
5. This .txt file or row in the DB contains the following: Your name, (billing) address, social security number, credit card number, mother’s maiden name and your bank’s ABA routing number and checking account number
6. I’ll invite some people I know to test your hypothesis for you

Let’s see if they want to put their money (literally) where their mouths are?  After all, they claim that Cloud providers will be able to secure their applications and data.

I triple-dog-dare you.

The only diatribes that we ought to be spared from are those that themselves don’t offer a balance of reality, responsibility and maturity as those they accuse of doing the same.

It’s not that Cloud deployments *can’t* be at least as secure as non-Cloud deployments with appropriate adjustments.  My issue with these wanderlust expressions is that the implication today that Cloud providers not only achieve parity but also exceed it — and that Cloud providers have some capability or technology the rest of us do not — given the challenges we have, is incredulous.

I’m all for evangelism, but generalizing about the state of security (in Cloud or otherwise) is a complete waste of electrons.  Yes, Cloud brings us opportunity and acts as a forcing function and we *will* see improvements, but NOT because we put blinders on and pretend that the delivery model (Cloud) will fix 40 years of legacy computing challenges — especially since Cloud is built upon most of them in the first place!

See here.

/Hoff

—

* Feel free to use SSL if it makes you feel any better.

My friend Adam at the The New School Information Security Blog wrote An Open Letter to the New Cyber-Security Czar:

Congratulations on the new job! Even as a cynic, I’m surprised at just how fast the knives have come out, declaring that you’ll get nothing done. I suppose that low expectations are easy to exceed. We both know you didn’t take this job because you expected it to be easy or fun, but you know better than most how hard it will be to make a difference without a budget or authority. You know about many of the issues you’ll need to work through, and I’d like to suggest a few less traditional things which you can accomplish that will help transform cyber-security.

Adam’s thoughtful post was chock full of interesting points and guidance associated with what he and others think Howard Schmidt ought to consider in his “new” role as Cyber-Security Coordinator.

My suggestion was a little more simple in nature:

November and December usually signal the onslaught of security predictions for the coming year. They’re usually focused on the negative.

I’ve done these a couple of times and while I find the mental exercise interesting, it really doesn’t result in anything, well, actionable.

So, this year I’m going to state what I am *going* to do rather than what I think others *might.*  I’ve spent the last couple of years talking about the challenges, now it’s time to focus on the solutions.

It’s quite simple.  I resolve to:

1. Continue my efforts to make the Cloud Security Alliance work products more useful and impactful, focusing on solutions to the challenges we have with Cloud Security
2. Push the agenda for transparency in Cloud providers with the A6 API working group
3. Deliver even more interesting and thought-provoking presentations focused on virtualization and Cloud security
4. Take our local security scene up a notch: focus on making BeanSec more than just a social event and make it the epicenter for security knowledge sharing in the greater Boston area
5. Spend more time at local events such as ISACA and OWASP and support regional “non-cons”; many folks don’t get to go to the big shows
6. Blog more and push the envelope on things I know need to improve.  Also publish the podcast and vlogs on a regular basis
7. Reach out beyond the U.S. and share more/learn more with folks from other countries/backgrounds
8. Dig my heels in and participate more actively in the standards bodies and organizations that I lurk in (PCI vSig, DMTF, etc.)
9. Focus on making my contacts into more of a community; I have the most awesome circle of friends and acquaintances and it’s time to put them to use
10. Publish a couple of the books I’m working on

These are my top 10.

What are yours?

/Hoff

Version 2.1 of the Cloud Security Alliance “Security Guidance for Critical Areas of Focus in Cloud Computing” is available for download here.

It’s important to note that in this version of the guidance there are some notable changes in structure and content focus:

The guidance provided herein is the second version of the Cloud Security Alliance document, “Security Guidance for Critical Areas of Focus in Cloud Computing”, which was originally released in April 2009.  The permanent archive locations for these documents are:

http://www.cloudsecurityalliance.org/guidance/csaguide.v2.1.pdf  (this document)
http://www.cloudsecurityalliance.org/guidance/csaguide.v1.0.pdf  (version 1 guidance)

In a departure from the first version of our guidance, a decision was made to separate the key guidance from the core domain research.  Each domain’s core research is being released as its own white paper.  These white papers and their release schedule are located at:

http://www.cloudsecurityalliance.org/guidance/domains/

In another change from the first version, Domain 3: Legal and Domain 4: Electronic Discovery were combined into a single domain.  Additionally, Domain 6: Information Lifecycle Management and Domain 14: Storage were combined into a single domain, renamed Data Lifecycle Management.  This has caused a renumbering of our (now 13) domains.

We have hundreds of pages of edited/compiled content for each of these domains and the working groups will be releasing their schedules for the domain work products shortly.

Thanks to everyone who contributed!  We look forward to delivering even more value in the follow-on releases.

/Hoff,
Technical Advisor CSA