Measuring Security Performance, Part II

As a follow-up to the “Measuring Security Performance” blog post I thought I’d spend some more time talking about my thoughts on performance metrics now that our Catalyst conference is over, my “Security KPIs” document has made it to our production group, and the Metricon 4.0 conference happened last week.

I had some very good audience questions during my Catalyst talk – one of which was from a business line manager who wondered how we should measure the “failings” of the security team that complicated the work day of his team. To me the answer to this doesn’t lie in the security metrics space (although we could certainly use them to look for the cause and effect once we know there’s an issue) but rather in general operational metrics such as help desk calls and bug tracking systems.  And this has to be one of our considerations when trying to measure the program: what are the neighboring processes, technologies and their metrics, and how can we leverage them to their best effect?

The question about what is already happening around us is of course part of the context in which we measure, and this is something I heavily focus on in the upcoming security KPIs document. Coincidentally, or perhaps in a stroke of luck, the theme of Metricon 4.0 was actually “The Importance of Context.” I consider contextualized measurement to be of great value for the program-level measurements: by defining units of measurement that are abstracted from the underlying implementation, we can more easily define the goals we’re looking to achieve. Measuring virus counts and patching speed is certainly interesting at the lower level, but at the mid-level we really need to think in terms of such concepts as risks, audit findings, and incidents, and how these relate in terms of performance.

Because enterprises have varying definitions of what constitutes an incident or a material risk this doesn’t necessarily provide us with an easy way to compare between organizations (benchmarking), but then again neither do measurements such as “security budget as percentage of IT budget” or “average time to patch.” But at least by creating an abstraction layer we should be able to better understand and communicate the things that really matter in terms that are understandable.

Comments

[Anne Stephanie Cruz]

Hi!

Just sending you a short note to say I really enjoy reading your blog.

I’m Steph Cruz and I work for a background screening service provider called Crimcheck.com. I was wondering if you would allow me to write a guest post for you. The article can be about HR, background checks, employment screening - or really any topic that you might find relevant.

Please let me know how this idea sounds to you. Thanks in advance!

Sincerely Yours,

Anne Stephanie Cruz
www.crimcheck.com

Steph@crimcheckadvantage.com