All Your Virtualized Storage Are Belong To Us…
A point I’ve raised in my Four Horsemen presentation that I’m spending a lot of time researching is on the topic of how virtualization is accelerating the adoption of SAN storage and to a larger extent I/O virtualization and what that means to security.
I wanted to introduce this topic as a teaser to a larger series of posts. This is not a treatise on the subject and is deliberately thin on detail and thick on corner case illustration. You have been warned.
What I’m both extremely intrigued by and really worried about as we move further along the virtualization continuum is the strange duality offered up by the intersection of consolidated compute, resources and information overlaid with tremendous distributed mobility and the security (or lack thereof) of this pooled storage holding all our crown jewels.
Centralized storage offers great benefits: easier to backup/restore, simpler management, better cost effectiveness, facilitated resiliency and potentially better security. However, that all depends upon how and who is actually responsible for the operational aspects of security and defining the policies and compliance requirements in the first place.
We see the grappling matches of responsibility being waged between who is responsible for securing our basic virtualized environments from the "server" and "network" perspective today and we’re struggling for answers.
What about securing storage?
This isn’t an argument about filesystem partitioning, ACL’s and GPO. This is a whole other networking fabric or set of appliances that are being deployed, conntected to our hosts and administered/secured by…who?
Think about it; we’re moving away from local storage to pooled "networked" storage. Not only is our critical information stored in these pools, but the Virtual Machine (VM) images are also. Databases too. One stop shopping!
Sure, this has trend has been going on for years, but virtualization and consolidation are shining the ugly light on the fact that we’ve been covering our eyes as to what this means to our overall security posture for many years. That’s not going to last much longer.
Depending upon who is responsible for the architecture of your virtualized storage, your perfectly reasonable asset, network segmentation and layered defense may go out the window when "machines" from multiple tiers all interconnect to a single storage fabric.
It shouldn’t happen, but it does; think about your (coming soon) virtualized DMZ’s and what that might mean to this diagram:
You might notice that for "simplicity" although the management/service console network is represented, the storage "network" is not. I’ll bet dollars to doughnuts that all three tiers would be serviced by a single SAN in the real world…
I’ve heard that over your dead body would you combine multiple network zones on the same physical/virtualized host like in the picture above. Strangely though I bet many of you connect physical assets (virtualized or not) of varying criticality to the same SAN fabric though like in the Brocade diagram on the top left.
What exactly is the difference?
Will the real SAN storage security experts please stand up? OK, how
about if you’re faking it really well? Um, how about you storage
admins posing as security experts? Server admins who eat LUNs for
lunch? Network engineers who admin Cisco, Brocade, Xsigo I/O virtualization switches?
Bueller?
I am *not* a security storage expert but I’m reasonably skilled in many
other elements of security, and that’s really the point of all of
this. I recognize there are many things that can be done to segment/zone/mask/secure
storage, but I don’t have those skills and I don’t know how to assess others’ either. I’d bet that if you haven’t
been involved in securing storage for quite some time, even if you’re
being drug into virtualized environments, you’re in the same boat?
Food for (scary) thought.
/Hoff
I think a couple of historical factors have lulled the storage security community to sleep. A couple of these factors include the lack of interoperability of SAN storage products, and the logical separation the layers of logical and physical separation that are inherent to most SAN designs. Separate FC infrastructure, with LUNS zoned and masked to be accessed by only one host (unless deploying a clustered solution, where the same LUNS may be shared) But as you've noted, many things are changing, that should bring about a rude awakening. The virtualization layer sees all in a shared storage solution and i'm not sure those in the storage industry really gets that yet. Add that to the proliferation of IP storage, and then sprinkle in a virtualized network layer, and you have a perfect storm of factors that multiplies risk exponentially. The security industry needs to come to term with the changing landscape, and step up to the plate.
I know enough to be dangerous, so I'll stick my neck out and kick off the thread.;)
In general –
– If you have access to the SAN switch fabric and the appliance/software/device that manages the controllers it's pretty much game over. With fabric administrative access, I could clone a LUN from one server and present it to another server (presumably another server in a lower security zone) using the standard management tools or with access to both the switch fabric (zoning) and the controllers (LUN management). In the particluar case that I am very familiar with, we can clone LUN from a server in one city and present it to a server in another city, all via the fiber channel fabric. (Cool, eh?)
– In some cases, in-band management is possible – i.e. a production server can clone it's own LUN's and present them to another server, or a server can have a management tookit loaded on it that allows it to manage LUN's on other servers, with authentication, of course.
– If the SAN is using WWID's for zoning, and if I could clone a WWID, I might be able to make bad things happen. (Still thinking about that one.)
– There are lots of options for AAA-like access controls on SAN devices. It'd be interesting to see how often they are used.
In other words, a SAN that crosses security zones really is not necessarily worse that a network fabric that crosses security zones. If a bad guy can mess with the fabric, bad things can happen.
The really interesting questions:
– do the dudes that manage the SAN think like paranoid network/security geeks? To they spend there spare time checking out the latest hack/tool/'sploit?
– Do the SAN fabric vendors think of the SAN equivalents to arp cache poisoning, cam table flooding, etc?
Probably not.
The DMZ diagram and associated comment are thought provoking. I'm no security expert, but it seems to me that if your VMs in your DMZs were encapsulating their storage through the virtual infrastructure and not accessing the shared storage directly (ie. only touching the SAN via their vmdk files, via the ESX servers and via a completely isolated storage network), there would be no way get to the storage of the other servers on the SAN even if one of the web servers was completely compromised. Of course, if your VMs had direct access to the SAN all bets are off.
Storage must be secure? Right? However, FC-SAN, iSCSI and NFS are cleartext protocols. Granted authentication for FC-SAN and iSCSI can be encrypted, but is that enough?
If IPv6 w/IPSec was used for iSCSI and NFS is that enough? But is this supported by virtualization servers today?
I see people setting up iSCSI servers that their VMs access and their virtualization servers access, is this protected?
Now introduce NPIV? Can all those VLAN attacks be used against NPIV.
More food for thought….
So glad someone is thinking about this specific issue. I don't have any solidified thoughts about how to address it but the conversation needs to start. SAN is just one place to start. There's lot of other resources the vm's will be dependent on that will provide opportunities as well.