This sixth episode of Take5 interviews Andy Jaquith, Yankee Group analyst and champion of all things Metric…I must tell you that Andy’s answers to my interview questions were amazing to read and I really appreciate the thought and effort he put into this.
First a little background on the victim:
Andrew Jaquith is a program manager in Yankee Group’s Enabling
Technologies Enterprise group with expertise in portable digital
identity and web application security. As Yankee Group’s lead security
analyst, Jaquith drives the company’s security research agenda and
researches disruptive technologies that enable tomorrow’s Anywhere
Enterprise™ to secure its information assets.
Jaquith
has 15 years of IT experience. Before joining Yankee Group, he
co-founded and served as program director at @stake, Inc., a security
consulting pioneer, which Symantec Corporation acquired in 2004. Before
@stake, Jaquith held project manager and business analyst positions at
Cambridge Technology Partners and FedEx Corporation.
His application security and metrics research has been featured in publications such as CIO, CSO and the IEEE Security & Privacy.
In addition, Jaquith is the co-developer of a popular open source wiki
software package. He is also the author of the recently released
Pearson Addison-Wesley book, Security Metrics: Replacing Fear, Uncertainty and Doubt. It has been praised by reviewers as both ”sparking and witty“ and ”one of the best written security books ever.” Jaquith holds a B.A. degree in economics and political science from Yale University.
Questions:
1) Metrics. Why is this such a contentious topic? Isn't the basic axiom of "you can't
manage what you don't measure" just common sense? A discussion on metrics evokes very
passionate discussion amongst both proponents and opponents alike. Why are we still
debating the utility of measurement?
The arguments over metrics are overstated, but to the extent they are
contentious, it is because "metrics" means different things to
different people. For some people, who take a risk-centric view of
security, metrics are about estimating risk based on a model. I'd put
Pete Lindstrom, Russell Cameron Thomas and Alex Hutton in this camp.
For those with an IT operations background, metrics are what you get
when you measure ongoing activities. Rich Bejtlich and I are
probably closer to this view of the world. And there is a third camp
that feels metrics should be all about financial measures, which
brings us into the whole "return on security investment" topic. A lot
of the ALE crowd thinks this is what metrics ought to be about. Just
about every security certification course (SANS, CISSP) talks about
ALE, for reasons I cannot fathom.
Once you understand that a person's point of view of "metrics" is
going to be different depending on the camp they are in -- risk,
operations or financial -- you can see why there might be some
controversy between these three camps. There's also a fourth group
that takes a look at the fracas and says, "I know why measuring
things matter, but I don't believe a word any of you are talking
about." That's Mike Rothman's view, I suspect.
Personally, I have always taken the view that metrics should measure
things as they are (the second perspective), not as you imagine,
model or expect them to be. That's another way of saying that I am an
epiricist. If you collect data on things and swirl them around in a
blender, interesting things will stratify out.
Putting it another way: I am a measurer rather than a modeler. I
don't claim to know what the most important security metrics are. But
I do know that people measure certain things, and that those things
gives them insights into their firms performance. To that end, I've
got about 100 metrics documented in my book; these are largely based
what people tell me they measure. Dan Geer likes to say, "it almost
doesn't matter what you measure, but get started and measure
something." The point of my book, largely, is to give some ideas
about what those somethings might be, and to suggest techniques for
analyzing the data once you have them.
Metrics aren't really that contentious. Just about everyone in the
securitymetrics.org community is pretty friendly and courteous. It's
a "big tent." Most of the differences are with respect to
inclination. But, outside of the "metrics community" it really comes
down to a basic question of belief: you either believe that security
can be measured or you don't.
The way you phrased your question, by the way, implies that you
probably align a little more closely with my operational/empiricist
view of metrics. But I'd expect that, Chris -- you've been a CSO, and
in charge of operational stuff before. :)
2) You've got a storied background from FedEx to @Stake to the Yankee Group.
I see your experience trending from the operational to the analytical. How much of your
operational experience lends itself to the practical collection and presentation of
metrics -- specifically security metrics? Does your broad experience help you in
choosing what to measure and how?
That's a keen insight, and one I haven't thought of before. You've
caused me to get all introspective all of a sudden. Let me see if I
can unravel the winding path that's gotten me to where I am today.
My early career was spent as an IT analyst at Roadway, a serious,
operationally-focused trucking firm. You know those large trailers
you see on the highways with "ROADWAY" on them? That's the company I
was with. They had a reputation as being like the Marines. Now, I
wasn't involved in the actual day-to-day operations side of the
business, but when you work in IT for a company like that you get to
know the business side. As part of my training I had to do "ride
alongs," morning deliveries and customer visits. Later, I moved to
the contract logistics side of the house, where I helped plan IT
systems for transportation brokerage services and contract warehouses
the company ran. The logistics division was the part of Roadway that
was actually acquired by FedEx.
I think warehouses are just fascinating. They are one hell of a lot
more IT intensive than you might think. I don't just mean bar code
readers, forklifts and inventory control systems; I mean also the
decision support systems that produce metrics used for analysis. For
example, warehouses measure an overall metric for efficiency called
"inventory turns" that describes how fast your stock moves through
the warehouse. If you put something in on January 1 and move it out
on December 31 of the same year, that part has a "velocity" of 1 turn
per year. Because warehouses are real estate like any other, you can
spread out your fixed costs by increasing the number of turns through
the warehouse.
For example, one of the reasons why Dell -- a former customer of mine
at Roadway-- was successful was that they figured out how to make
their suppliers hold their inventory for them and deliver it to final
assembly on a "just-in-time" (JIT) basis, instead of keeping lots of
inventory on hand themselves. That enabled them to increase the
number of turns through their warehouses to something like 40 per
year, when the average for manufacturing was like 12. That efficiency
gain translated directly to profitability. (Digression: Apple, by the
way, has lately been doing about 50 turns a year through their
warehouses. Any wonder why they make as much money on PCs as HP, who
has 6 times more market share? It's not *just* because they choose
their markets so that they don't get suckered into the low margin
part of the business; it's also because their supply chain operations
are phenomenally efficient.)
Another thing I think is fascinating about warehouses is how you
account for inventory. Most operators divide their inventory into
"A", "B" and "C" goods based on how fast they turn through the
warehouse. The "A" parts might circulate 10-50x faster than the C
parts. So, a direct consequence is that when you lay out a warehouse
you do it so that you can pick and ship your A parts fastest. The
faster you do that, the more efficient your labor force and the less
it costs you to run things. Neat, huh?
Now, I mention these things not strictly speaking to show you what a
smartypants I am about supp
ly chain operations. The real point is to
show how serious operational decisions are made based on deep
analytics. Everything I just mentioned can be modeled and measured:
where you site the warehouses themselves, how you design the
warehouse to maximize your ability to pick and ship the highest-
velocity items, and what your key indicators are. There's a virtuous
feedback loop in place that helps operators understand where they are
spending their time and money, and that in turn drives new
innovations that increase efficiencies.
In supply chain, the analytics are the key to absolutely everything.
And they are critical in an industry where costs matter. In that
regard, manufacturing shares a lot with investment banking: leading
firms are willing to invest a phenomenal amount of time and money to
shave off a few basis points on a Treasury bill derivative. But this
is done with experiential, analytical data used for decision support
that is matched with a model. You'd never be able to justify
redesigning a warehouse or hiring all of Yale's graduating math
majors unless you could quantify and measure the impact of those
investments on the processes themselves. Experiential data feeds, and
improves, the model.
In contrast, when you look at security you see nothing like this.
Fear and uncertainty rule, and almost every spending decision is made
on the basis of intuition rather than facts. Acquisition costs
matter, but operational costs don't. Can you imagine what would
happen if you plucked a seasoned supply-chain operations manager out
of a warehouse, plonked him down in a chair, and asked him to watch
his security counterpart in action? Bear in mind this is a world
where his CSO friend is told that the answer to everything is "buy
our software" or "install our appliance." The warehouse guy would
look at him like he had two heads. Because in his world, you don't
spray dollars all over the place until you have a detailed, grounded,
empirical view of what your processes are all about. You simply don't
have the budget to do it any other way.
But in security, the operations side of things is so immature, and
the gullibility of CIOs and CEOs so high, that they are willing to
write checks without knowing whether the things they are buying
actually work. And by "work," I don't mean "can be shown to stop a
certain number of viruses at the border"; I mean "can be shown to
decrease time required to respond to a security incident" or "has
increased the company's ability to share information without
incurring extra costs," or "has cut the pro-rata share of our IT
operations spent on rebuilding desktops."
Putting myself in the warehouse manager's shoes again: for security,
I'd like to know why nobody talks about activity-based costing. Or
about process metrics -- that is, cycle times for everyday security
activities -- in a serious way. Or benchmarking -- does my firm have
twice as many security defects in our web applications as yours? Are
we in the first, second, third or fourth quartiles?
If the large security companies were serious, we'd have a firmer grip
on the activity, impact and cost side of the ledger. For example, why
won't AV companies disclose how much malware is actually circulating
within their customer bases, despite their promises of "total
protection"? When the WMF zero-day exploit came out, how come none of
the security companies knew how many of their customers were
infected? And how much the cleanup efforts cost? Either nobody knew,
or nobody wanted to tell. I think it's the former. If I were in the
shoes of my Roadway operational friend, I'd be pissed off about the
complete lack of feedback between spending, activities, impact and cost.
If this sounds like a very odd take on security, it is. My mentor and
former boss, Dan Geer, likes to say that there are a lot of people
who don't have classical security training, but who bring "hybrid
vigor" to the field. I identify with that. With my metrics research,
I just want to see if we can bring serious analytic rigor to a field
that has resisted it for so long. And I mean that in an operational
way, not a risk-equation way.
So, that's an exceptionally long-winded way of saying "yes" to your
question -- I've trended from operational to analytical. I'm not sure
that my past experience has necessarily helped me pick particular
security metrics per se, but it has definitely biased me towards
those that are operational rather than risk-based.
3) You've recently published your book. I think it was a great appetite whetter but
I was left -- as were I think many of us who are members of the "lazy guild"-- wanting
more. Do you plan to follow-up with a metrics toolkit of sorts? You know, a templated
guide -- Metrics for Dummies?
You know, that's a great point. The fabulous blogger Layer 8, who
gave my book an otherwise stunning review that I am very grateful for
("I tucked myself into bed, hoping to sleep—but I could not sleep
until I had read Security Metrics cover to cover. It was That
Good."), also had that same reservation. Her comment was, "that final
chapter just stopped short and dumped me off the end of the book,
without so much as a fare-thee-well Final Overall Summary. It just
stopped, and without another word, put on its clothes and went home".
Comparing my prose to a one night stand is pretty funny, and a
compliment.
Ironically, as the deadline for the book drew near, I had this great
idea that I'd put in a little cheat-sheet in the back, either as an
appendix or as part of the endpapers. But as with many things, I
simply ran out of time. I did what Microsoft did to get Vista out the
door -- I had to cut features and ship the fargin' bastid.
One of the great things about writing a book is that people write you
letters when they like or loathe something they read. Just about all
of my feedback has been very positive, and I have received a number
of very thoughtful comments that shed light on what readers'
companies are doing with metrics. I hope to use the feedback I've
gotten to help me put together a "cheat sheet" that will boil the
metrics I discuss in the book into something easier to digest.
4) You've written about the impending death of traditional Anti-Virus technology and its
evolution to combat the greater threats from adaptive Malware. What role do you think
virtualization technology that provides a sandboxed browsing environment will have in
this space, specifically on client-side security?
It's pretty obvious that we need to do something to shore up the
shortcomings of signature-based anti-
malware software. I regularly
check out a few of the anti-virus benchmarking services, like the
OITC site that aggregates the Virustotal scans. And I talk to a
number of anti-malware companies who tell me things they are seeing.
It's pretty clear that current approaches are running out of gas. All
you have to do is look at the numbers: unique malware samples are
doubling every year, and detection rates for previously-unseen
malware range from the single digits to the 80% mark. For an industry
that has long said they offered "total protection," anything less
than 100% is a black eye.
Virtualization is one of several alternative approaches that vendors
are using to help boost detection rates. The idea with virtualization
is to run a piece of suspected malware in a virtual machine to see
what it does. If, after the fact, you determine that it did something
naughty, you can block it from running in the real environment. It
sounds like a good approach to me, and is best used in combination
with other technologies.
Now, I'm not positive how pervasive this is going to be on the
desktop. Existing products are already pretty resource-hungry.
Virtualization would add to the burden. You've probably heard people
joke: "thank God computers are dual-core these days, because we need
one of 'em to run the security software on." But I do think that
virtualized environments used for malware detection will become a
fixture in gateways and appliances.
Other emergent ideas that complement virtualization are behavior
blocking and herd intelligence. Herd intelligence -- a huge malware
blacklist-in-the-sky -- is a natural services play, and I believe all
successful anti-malware companies will have to embrace something like
this to survive.
5) We've see the emergence of some fairly important back-office critical applications
make their way to the Web (CRM, ERP, Financials) and now GoogleApps is staking a hold
for the SMB. How do you see the SaaS model affecting the management of security --
and ultimately risk -- over time?
Software as a service for security is already here. We've already
seen fairly pervasive managed firewall service offerings -- the
carriers and companies like IBM Global Services have been offering
them for years. Firewalls still matter, but they are nowhere near as
important to the overall defense posture as before. That's partly
because companies need to put a lot of holes in the firewall. But
it's also because some ports, like HTTP/HTTPS, are overloaded with
lots of other things: web services, instant messaging, VPN tunnels
and the like. It's a bit like the old college prank of filling a
paper bag with shaving cream, sliding it under a shut door, then jumping
on it and spraying the payload all over the room's occupants. HTTP is
today's paper bag.
In the services realm, for more exciting action, look at what
MessageLabs and Postini have done with the message hygiene space. At
Yankee we've been telling our customers that there's no reason why an
enterprise should bother to build bespoke gateway anti-spam and anti-
malware infrastructures any more. That's not just because we like
MessageLabs or Postini. It's also because the managed services have a
wider view of traffic than a single enterprise will ever have, and
benefit from economies of scale on the research side, not to mention
the actual operations.
Managed services have another hidden benefit; you can also change
services pretty easily if you're unhappy. It puts the service
provider's incentives in the right place. Qualys, for example,
understands this point very well; they know that customers will leave
them in an instant if they stop innovating. And, of course, whenever
you accumulate large amounts of performance data across your customer
base, you can benchmark things. (A subject near and dear to my heart,
as you know.)
With regards to the question about risk, I think managed services do
change the risk posture a bit. On the one hand, the act of
outsourcing an activity to an external party moves a portion of the
operational risk to that party. This is the "transfer" option of the
classic "ignore, mitigate, transfer" set of choices that risk
management presents. Managed services also reduce political risk in a
"cover your ass" sense, too, because if something goes wrong you can
always point out that, for instance, lots of other people use the
same vendor you use, which puts you all in the same risk category.
This is, if you will, the "generally accepted practice" defense.
That said, particular managed services with large customer bases
could accrue more risk by virtue of the fact that they are bigger
targets for mischief. Do I think, for example, that spammers target
some of their evasive techniques towards Postini and MessageLabs? I
am sure they do. But I would still feel safer outsourcing to them
rather than maintaining my own custom infrastructure.
Overall, I feel that managed services will have a "smoothing" or
dampening effect on the risk postures of enterprises taken in
aggregate, in the sense that they will decrease the volatility in
risk relative to the broader set of enterprises (the "alpha", if you
will). Ideally, this should also mean a decrease in the *absolute*
amount of risk. Putting this another way: if you're a rifle shooter,
it's always better to see your bullets clustered closely together,
even if they don't hit near the bull's eye, rather than seeing them
near the center, but dispersed. Managed services, it seems to me, can
help enterprises converge their overall levels of security -- put the
bullets a little closer together instead of all over the place.
Regulation, in cases where it is prescriptive, tends to do that too.
Bonus Question:
6) If you had one magical dashboard that could display 5 critical security metrics
to the Board/Executive Management, regardless ofindustry, what would those elements
be?
I would use the Balanced Scorecard, a creation of Harvard professors
Kaplan and Norton. It divides executive management metrics into four
perspectives: financial, internal operations, customer, and learning
and growth. The idea is to create a dashboard that incorporates 6-8
metrics into each perspective. The Balanced Scorecard is well known
to the corner office, and is something that I think every security
person should learn about. With a little work, I believe quite
strongly that security metrics can be made to fit into this framework.
Now, you might ask yourself, I've spent all of this work organizing
my IT security policies along the lines of ISO 17799/2700x, or COBIT,
or ITIL. So why can't I put together a dashboard that organizes the
measurements in those terms? What's wrong with the frameworks I've
been using? Nothing, really, if you are a security person. But i
f you
really want a "magic dashboard" that crosses over to the business
units, I think basing scorecards on security frameworks is a bad
idea. That's not because the frameworks are bad (in fact most of them
are quite good), but because they aren't aligned with the business.
I'd rather use a taxonomy the rest of the executive team can
understand. Rather than make them understand a security or IT
framework, I'd rather try to meet them halfway and frame things in
terms of the way they think.
So, for example: for Financial metrics, I'd measure how much my IT
security infrastructure is costing, straight up, and on an activity-
based perspective. I'd want to know how much it costs to secure each
revenue-generating transaction; quick-and-dirty risk scores for
revenue-generating and revenue/cost-accounting systems; DDOS downtime
costs. For the Customer perspective I'd want to know the percentage
and number of customers who have access to internal systems; cycle
times for onboarding/offloading customer accounts; "toxicity rates"
of customer data I manage; the number of privacy issues we've had;
the percentage of customers who have consulted with the security
team; number and kind of remediation costs of audit items that are
customer-related; number and kind of regulatory audits completed per
period, etc. The Internal Process perspective has of the really easy
things to measure, and is all about security ops: patching
efficiency, coverage and control metrics, and the like. For Learning
and Growth, it would be about threat/planning horizon metrics,
security team consultations, employee training effectiveness and
latency, and other issues that measure whether we're getting
employees to exhibit the right behaviors and acquire the right skills.
That's meant to be an illustrative list rather than definitive, and I
confess it is rather dense. At the risk of getting all Schneier on
you, I'd refer your readers to the book for more details. Readers can
pick and choose from the "catalog" and find metrics that work for
their organizations.
Overall, I do think that we need to think a whole lot less about
things like ISO and a whole lot more about things like the Balanced
Scorecard. We need to stop erecting temples to Securityness that
executives don't give a damn about and won't be persuaded to enter.
And when we focus just on dollars, ALE and "security ROI", we make
things too simple. We obscure the richness of the data that we can
gather, empirically, from the systems we already own. Ironically, the
Balanced Scorecard itself was created to encourage executives to move
beyond purely financial measures. Fifteen years later, you'd think we
security practitioners would have taken the hint.
flawed, and at worst ineffective. Examples include:
While
Recent Comments