Hey, Uh, Someone Just Powered Off Our Firewall Virtual Appliance…
I’ve covered this before in more complex terms, but I thought I’d reintroduce the topic due to a very relevant discussion I just had recently (*cough cough*)
So here’s an interesting scenario in virtualized and/or Cloud environments that make use of virtual appliances to provide security capabilities*:
Since virtual appliances (VAs) are just virtual machines (VMs) what happens when a SysAdmin spins down or moves one that happens to be your shiny new firewall protecting your production VMs behind it, accidentally or maliciously? Brings new meaning to the phrase “failing closed.”
Without getting into the vagaries of vendor specific mobility-enabled/enabling technologies, one of the issues with VMs/VAs is that there’s not really a good way of designating one as being “more important” or functionally differentiated such as “security” or “critical application” that would otherwise ensure a higher priority for service availability (read: don’t spin this down unless…) or provide a topological dependency hierarchy in virtualized network constructs.
Unlike physical environments where system administrators (servers) are segregated from access to network and security appliances, this isn’t the case in virtual environments. In Cloud environments (especially public, multi-tenant) where we are often reliant only upon virtual security capabilities since we have no option for physical alternatives, this is an interesting corner case.
We’ve talked a lot about visibility, audit and policy management in virtual environments and this is a poignant example.
/Hoff
*Despite the silly notion that the Google dudes tried to suggest I equated virtualization with Cloud as one-in-the-same, I don’t.
Chris, prioritization and dependency checking should be part of any orchestration process, right? An organization should, either through product or process, be able to take steps to ensure critical services are available and take them down in a controlled manner. At least, I would hope so.
Seems like one of those cases where systems managment is the same in the virtual world as in the physical.
@mike
SOMEBODY'S paying attention 😉 You bet. That's part of the problem; the orchestration (and governance and provisioning…) systems are still decoupled from much of the topological dependencies — at the lower AND higher application layers.
Fun stuff that. I'm not sure we've got it that well handled in the physical world — it occurs to me that the manual silos/separation of duties couches some of these issues because it's NOT automated/autonomic and so easily provisioned.
Did that make sense?
/Hoff
Good point. One of the tenets in computer security states that if the bad guys have physical access, all bets are off. But in a virtual environment, it turns out "physical access" to a VM no longer requires physical access to the real hardware…
Precisely. When you have clicky-clicky-VM-admins in charge *who no longer know the VMs the way OS admins do,* you have a much greater risk of having critical ones taken down because they just didn't know better. Unless the hostname is BOOTMEFIRST or DONTTAKETHISDOWNFORGODSSAKE. 😉
Cloud administration knowledge is no substitute for business/architecture knowledge. The latter is what the old sk00l sysadmins had and it's what you're giving up when you cloudsource.
Oh, and get off my lawn.
In the case of VMware DRS, one has the ability to pin VMs to specific machines, and to you also have the ability to specify specific resources for the machines as well. I imagine you can also tie this in with secure access to the VC orchestrating the entire operation to get somewhat close to what you want above. I think any decent virtualization solution needs to have the ability to do what you say above.
Good post. Definitely a thing to think about and focus on in virtual security topologies. For now this is why we sell VPN-Cubed managers in "pairs", so that people configure a back up VM as the default topology; since hardware fails, VMs fail.
And even when you have the CMDB going down to VM and application level, there's still the need to put the data itself into the model. Things won't work without those crypto keys.
As Daniel Dresner put it in http://www.nacr.cz/dlm/presentations/dresdner.pdf
2008 – The year of lost data (UK)
2009 – The year of encryption
2010 – The year of lost encryption keys
Great timing for that post Hoff. We just setup vShield Zones in our lab the other day and deployed a vShield to the management vSwitch, which happens to be where our vCenter server and other important VMs live. After the 'automagic' deployment, we powered down the vShield to adjust the memory allocation, and found that as soon as the power went out, so did our connection to the vCenter box.
Process is likely going to be the best way to manage this new twist on a conventional risk. However, it would be nice if our tools supported marking certain VMs critical/immutable/etc.
Hoffmeister,
Great points, it highlights that virtualisation certainly has an awareness gap with security chaps. It also highlights we have a long way to go until the likes of EMC think we can have a completely virtualised datacentres with highly valid non starter topics like this being around.
Strategy is available to negate any issues occuring, such as providing delegated control to certain personel, siloing virtual infrastructure (most robust for DMZ hosts certainly). Also Technology from both a functionality and educational standpoint has to grow in maturity to support virtualised Security Appliances, maybe this is where the virtualisation vendors have to do the hardwork to ensure they dont get a bad name and work with security providers to ensure that best practices for running such virtualised security VM's is implemented correctly.
Perhaps the virtualisation industry even needs a educational track in either say the Cisco or VMware certification track to ensure virtualising such environments can be acheived.
Dan
PS I won't mention vShield Zones this takes this statement as with DC's experiences to another confusing level!
@DC
> After the ‘automagic’ deployment, we powered down the vShield to adjust the memory allocation, and found that as soon as the power went out, so did our connection to the vCenter box.
isn’t this what a customer may want anyway, i.e., fail-close — you don’t want your vCenter to be vulnerable when the FW is in maintenance mode)? i understand ideally you should have a choice, whether fail-open or fail-close.
If you are using VSphere, you should be looking at what Check Point's VE product will offer end of 09. Should pretty much take care of the scenario above.