Rational Survivability

Hey beaker,

I fully support your idea that devops is a multi-disciplinary approach so indeed should include other fields such as security and network. In several companies they fall under operations group as well, not under the sysadmins group. Therefore I welcome posts such as yours to create an opening to other groups. Both in the puppet and the chef community people are looking to extend their recipes to non-server machines such as switches, firewalls and so on.

Looks to me very similar to the dba's who slowly are getting integrated in 'Agile development'.

Personally I think it would be even better to have business too in the name: making it busidevops. But somehow devops as a term catches on. But that's another story 🙂

Let's hope this helps…

I can see how the messaging of the devops thoughts would seem missing something, because the idea is not about automation. It's actually not about a bunch of things:

It's not abstraction.

It's not even "infrastructure as code".

It's not any single tool.

It's not about provisioning.

It's not about deployment.

It's not about a job description or position.

It's also not about the cloud, except for the part where deployment and provisioning of infrastructure gets easier to understand for groups of people who historically wouldn't have touched that part of the business.

It *is* about the collaborative and communicative culture and the tools and process that arise from that culture. Nothing more.

But there's only so much that 140 characters can point out at a time, and what you're absorbing is snippets of a passionate group of people who are getting a glimpse of what happens when people actually talk to each other instead of framing the discussion as an "Us versus Them" position.

How you're understanding that the ideas surrounding would leave *any* part of an organization out of the loop escapes me, actually. The idea(s) behind 'devops' is explicitly to include people who enable the business, and to capture what each group can bring to the table, not to lock anyone out.

I don't care about the name, because frankly it could be ProdCareDesignBizNetSecDevEngOps as far as I'm concerned.

One last point…this here:

"In many presentations promoting DevOps, developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages."

is not personally my position. My position is that both dev and ops has allowed contention, opposition, and stereotyped silo-ing to happen as an accepted situation, and that 'devops' or whatever you want to call it…is the adjustment of those cultures and attitudes.

Successful engineering organizations bridge these gaps and work together. This means specifically not being fingerpointy, and not being exclusionary.

But again, I have a feeling it'll be easier to talk about it over beer. You'll be at Velocity?

I wouldn't assume that deeply committed devops orgs dont' include network device automation. I can't speak to the manual security processes, but certainly for things like network provisioning and basic policy the NASA Nebula project contemplates full layer 2 isolation orchestration using Chef.

"There's a recipe for that" (esp if you are Joshua McKenty and just write it yourself)

Chris,

It was a 5-minute lightning talk on how our product "Opscode Chef" fits in the fabric of a new way to treat infrastructure. It was a targeted presentation for our niche audience. It was not meant to be a “Devops Manifesto”. Where I totally agree with a lot of your points about devops including security, network (and you forgot Storage), using our slides as the "Devops" manifesto example is waaaaay out of context. As for the load balancer example, it's just an example. We also didn't mention the entire network and storage infrastructure between those components. That doesn't mean that we don't believe they exist. In fact we have many examples were the load balancer is API based. The Chef product is completely API based and in fact we use many scenarios where the resources we address are API abstractions (e.g., EC2, EBS,DNS).

Again, I agree that the Devops messaging needs to be fine-tuned in fact very similar to the NoSQL movement punted to “Not Only”. However, like NoSQl, I believe Devops” was a great stake in the ground to promote change. I think your voice is a much-needed one in this movement and you know I appreciate you being involved.

One more point for the record, I do not believe “Operations as Code”/”Infrastructure as Code” is a zero sum gain. Nor do I agree that it is the single based definition of “Devops”. In my heart of hearts, I believe Devops is about a “change”. Doing things differently from the way the old legacy infrastructure and following some of the exciting things going on in the new “Internet” positioned enterprises.

Hope that Helps

John Willis

@John Allspaw

John,

"It *is* about the collaborative and communicative culture and the tools and process that arise from that culture. Nothing more. "

Totally agree with this as one of the best definitions of Devops. When I try to position Opscode Chef I am doing so as one of the really cool tools that is trying to help with these cultural process changes. Not as the "definition" of devops. I have also been guilty of creating slide decks that show some of the older non agile practices of operations. The point I am trying to make there is that sys-admins that hoard scripts and companies that don't embrace operations as code might not fit into a model of highly scalable operations.

Thanks

John W.

I like the term "system development" – that's what our team was called in one of my former lives. That team consisted of dbas, system engineers and various other types. And we lived on the same floor as our developers …

I think it's important to separate out the two concepts at play here, that so

often get interwined – especially when you're talking to people from Opscode,

because we are so firmly entrenched in both. The first is the cultural and

professional shift towards Webops culture becoming the dominant Operations

culture, whcih people are calling DevOps. The second is the 'Infrastructure As

Code' phenomen, which also comes out of Webops culture, which is about building

and managing infrastructure programatically. While the two are often related,

they aren't the same thing.

I've always been uncomfortable with the word DevOps, for reasons similar to

you. For me, it's a free-pass to wallow in the second-class citizen status

many Systems Administrators have always felt. If I had a nickle for every

person I've known in this industry who believed they were less skilled or

valuable than a software developer, I would be a rich man. When I first

started to see the word tossed around, one of the most common contexts was as a

new job description.. and it drives me crazy. I'm proud to be a Systems

Administrator, and I'm proud to have peers like John Allspaw and Theo

Schlossnagle.

The messaging problem you point out is a real one – you can either talk about

what's happening culturally from a place of inclusion or a place of exclusion.

The dominant cultural paradigm remains one of silo-ed specialization, and it's

obvious from this post. As a Network and Security Administrator, you feel

excluded by the focus on Systems Operations – because you see the disciplines

are being separate. The shift here is that your specialization becomes

subserviant to the overall business objectives and efficiency – rather than

remaining separate from your peers, both organizationally and culturally, you

become integrated into the whole, and valued for your expertise (rather than

your organizational kingdom-building prowess).

The inclusive way to talk about this is to say that all of us play an integral

role in the success of our busineses. By coming together, communicating

clearly, and helping each other understand exactly what it is that we do, we

build more effective, efficient, and quite frankly *funner* teams and

companies. We stop focusing on the things that make us different, and we start

focusing on the way we compliment each other, and the ways we can help make

each others lives easier. You can spot this camp because they spend time

talking about how great their lives are, and how awesome what they build is.

The exclusive way to talk about it is to make it a private club. There are the

'enlightened' DevOps, and everyone else stuck living in the

technological equivilant of Plato's Cave. This is the camp that wants a new

job title – it sets you apart from the "others", who don't operate the way you

do. The messages are deceptively similar – they both talk about integrated

teams of Software Developers, and Systems/Network Administrators. You can

spot this by tracking the negativity – if you're talking about how much

something sucks, you're probably in this camp, rather than in the inclusive

one.

Devops, the cultural movement, is an awesome one. It's empowering and it makes

the lives of the people it influences exponentially better. Let's cheer on the

inclusionists and enlighten the exclusionists. 🙂

Okay, lets move to Infrastructure as Code. Where Devops is a cultural change,

this is a technological one – it's a shift in the way we think about assembling

the infrastructures that run our businesses. In my contribution to the fine

Mr. Allspaw's upcoming book, I define it through it's primary goal:

"Enable the reconstruction of the business from nothing but a source code

repository, an application data backup, and bare metal resources."

Secondarily, there is another ideal here – the thing that should take the

longest in this recovery process is the time it takes to restore your

application data.

This ideal isn't new – it tracks back at least as far as the
http://www.infrastructures.org guys. The modern version draws on Service

Oriented Architecture (and Web 2.0), Configuration Management, and Systems

Integration for inspiration. The arrival of a new breed of infrastructure

services (which in my mind started with EC2, honestly) lowered the barrier to

actually achieving the goal (partially by narrowing your options at the

physical layer, but primarily by giving us a real API to bootstrapping, which

for most of us, was the thing that took longest during recovery.)

Security, Networking, etc are all included here – if your goal is to

re-constitute your business from scratch, you better be able to take it all the

way. 🙂

When people give presentations on building Infrastructure as Code, it's no

suprise they focus on what can be easily automated. Vendors like Cisco and F5

have not been building systems that are easy to automate – quite the opposite.

When the APIs do exist, they are irritating to use, and difficult to integrate

with. (In the case of Cisco, there is another problem – the automation often

tries to abstract away the specialized knowledge embodied in every network

engineer, which frustates the hell out of them.)

We are still playing catch-up with physical infrastructure and proprietary

devices. As the Devops cultural shift gains momentum, and more examples of

fully-automated infrastructures exist in the world, we'll see the hurdles to

this approach in the traditional infrastructure drop. Vendors will see the

light, or the tradional enterprise will abandon their technology as fast as the

big web shops have.

I could go on all day, but I need to go for a run and burp my newborn baby. 🙂

Thanks for the post, and I hope this clears something up – at the very least, I

hope it underscores the dividing line between the cultural movement (Devops)

and the technological one (Infrastructure as Code). Thanks for the post, and lets talk at Velocity about how we can get the Security and Network community involved in both the cultural and technological movements that are afoot.

f5 -> devcentral, icontrol, tm shell

cisco -> netomata config generator (ncg)

Great post and comment thread. To me the names don't matter – DevOps/OpsDev/WebOps etc. The message we need to be spreading is about organisational cultural change in technology business units – not Ops, Dev, SysAdmin people changing but organisations changing together. Working collaboratively and efficiently to deliver results customers want and return value to the company (shareholders, etc).

I come from a largely non-WebOps background – large scale banking and finance ranging back to mainframe days – and we suffer probably more than any other technology businesses from the failings that DevOps is trying to combat. I, and others in our space, are working hard to spread the message outside the WebOps/Web 2.0 world, where a little of the light has already broken through, to people who are totally in the dark and really hurting because of a lack of collaboration and communication and a failure to embrace the needed cultural change.

I've got a on what DevOps means to me and some slides I recently presented at DevOpsDownUnder talking about what DevOps means and how to start making some of the difficult and needed cultural and technological changes.

In spite of it not being about finger pointing, I think if we (operations) are honest with ourselves we'd have to admit that we have been somewhat stuck in the dark ages.

But they *do* seem to have had more resources thrown at them in terms of figuring out how to improve that. These agile methodologies did come out of development practices after all. I'm frankly excited to see ops guys being honest about where we have been (or are), and more than that have ideas for how to make it better.

Maybe it's just me, but I've never felt like it was finger pointing, but rather our finally talking about that stupid elephant in the room. Everyone was thinking it, but people are finally talking about it. Dev isn't this shining example on a hill, but the work they've done to look at things holistically, including everyone in the process, has inspired us to find our way forward down a similar path. You can't know where to go next if you don't understand where you are now – and maybe how you got here.

I think those aspects of these presentations are valuable. We've lied to to ourselves for a long time, telling ourselves everything was fine, it was these constant changes from dev that were the problem. How many times have we said (even jokingly) "everything would be great if it wasn't for the (users|developers)". We can be so resistant to change that before we're shown the better path we really need to know why it's necessary.

Sorry I'm coming to this party a little late… Some good points, a really good article and some really good comments. I also agree that the meaning of DevOps has been unclear and not communicated well. At Opscamp Austin, even among the "plugged in to DevOps" people there was clearly some differing expectations (It's automation! It's collaboration! It's ops folks becoming developers! It's a sandwich!). I personally think that the confusion is because DevOps is a rich term covering a lot of stuff, and its alternate name of "agile infrastructure" or "agile operations" is better because it breaks down into the same kinds of areas as agile development – I try to delineate the different areas of concern within DevOps here…

I don't think that "DevOps" is an exclusionary term – Ops contains network, security, etc. just as Dev contains all the different varieties of coders (and testers and…). It just reflects that in every shop there are considered to be two "sides of the house" – the code/software people and the infrastructure/operations people, and that schism is harmful. Now, I do think that the term "Ops" is being used to cover two roles that are different in larger organizations, the "system admin/engineer" role that builds infrastructure and the "operations/support role" that maintains them, and that's an area to delve more into. Lack of cooperation among groups on that "side of the house" is a chronic problem. I don't think it's an either-or between devs understanding ops and writing their code to help them and automating operations, those are both parts of the collaboration and maturation process.

Now, the cloud environment does make it EASIER to accomplish agile operations, for the reasons you note – less crufty hardware layers to deal with and more standard APIs means you can automate with less effort (and in fact, because of the dynamic nature of the cloud, you are FORCED to automate; in traditional environments automation is always that nice-to-have you maybe eventually get around to).

Hi,

I totally agree that there is no clear or single definition of devops. First I thought a definition would be needed and I even thought about starting one in Wikipedia, but I didn’t. And from day to day I’m getting surer there is no definition needed at this time. Why should we limit devops with a definition? In my opinion devops just starts and by limiting it with a definition, it would not be as useful as it could be.

The current discussion about a definition of devops seems a bit of a black or white discussion to me. But why do we need such a narrow definition and do not understand the different points of view as different parts or stages (people, technology, culture, …) of the same thought?

Today devops seems to be mostly driven by webshops, but why limit it to them? Devops can also be applied to larger enterprises. It might get a bit trickier and take longer, because of more silos, more heterogeneous systems and legacy systems, but it is not impossible. This would be one more reason to not make the definition of devops to narrow. In enterprises it might be easier to apply devops one step after the other. The same happened with agile development. Most enterprises did a first small project with agile methods and than moved on. Why should we try a big-bang migration with devops?

Even the discussion about the name devops is unnecessary. The name does not count, it is the idea behind it. The name is short, you can easily remember it and pronounce it the right way. What else do you want to have? You can’t put the description of a whole idea or movement in just a single word.

Christian

Hey, some more recent news on this – I saw a great presentation recently on incorporating security into agile development teams that made me think of this article and "NetSecDevOps." I blogged it up and linked to the slides… http://theagileadmin.com/2010/09/02/devops-and-se…