As a follow-up to the “Measuring Security Performance” blog post I thought I’d spend some more time talking about my thoughts on performance metrics now that our Catalyst conference is over, my “Security KPIs” document has made it to our production group, and the Metricon 4.0 conference happened last week.
I had some very good audience questions during my Catalyst talk – one of which was from a business line manager who wondered how we should measure the “failings” of the security team that complicated the work day of his team. To me the answer to this doesn’t lie in the security metrics space (although we could certainly use them to look for the cause and effect once we know there’s an issue) but rather in general operational metrics such as help desk calls and bug tracking systems. And this has to be one of our considerations when trying to measure the program: what are the neighboring processes, technologies and their metrics, and how can we leverage them to their best effect?
The question about what is already happening around us is of course part of the context in which we measure, and this is something I heavily focus on in the upcoming security KPIs document. Coincidentally, or perhaps in a stroke of luck, the theme of Metricon 4.0 was actually “The Importance of Context.” I consider contextualized measurement to be of great value for the program-level measurements: by defining units of measurement that are abstracted from the underlying implementation, we can more easily define the goals we’re looking to achieve. Measuring virus counts and patching speed is certainly interesting at the lower level, but at the mid-level we really need to think in terms of such concepts as risks, audit findings, and incidents, and how these relate in terms of performance.
Because enterprises have varying definitions of what constitutes an incident or a material risk this doesn’t necessarily provide us with an easy way to compare between organizations (benchmarking), but then again neither do measurements such as “security budget as percentage of IT budget” or “average time to patch.” But at least by creating an abstraction layer we should be able to better understand and communicate the things that really matter in terms that are understandable.
Comments