# Metrics Revisited Kim L. Jones CISM, CISSP, CRISC, MSIA.

## Presentation on theme: "Metrics Revisited Kim L. Jones CISM, CISSP, CRISC, MSIA."— Presentation transcript:

Metrics Revisited Kim L. Jones CISM, CISSP, CRISC, MSIA

2 Sources and Inspirations Paul Glen, How to Speak to the Business –www.leadinggeeks.comwww.leadinggeeks.com Lance Hayden, IT Security Metrics: A Practical Framework for Measuring Security & Protecting Data Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt Kim L. Jones CISM, CISSP, CRISC, MSIA

3 The Mantra: “Infosec is Terrible at Metrics” The metrics we can measure has little to do with security –Ex: Success of Antivirus System The stuff we really need to convey is the hardest to collect/quantify –“What is the sound of one hand clapping?” When we quantify numbers, they question our calculations They really don’t care about security…only compliance –“What needs fixing in security, and when will it be fixed?” Kim L. Jones CISM, CISSP, CRISC, MSIA

4 Defining the Problem Good vs. Bad Metrics Contraxioms Asking the Right Question Kim L. Jones CISM, CISSP, CRISC, MSIA

5 Good vs. Bad Metrics Consistently Measured Cheap to Gather –Technologically driven, where possible Expressed as a cardinal number or a percentage Expressed using at least one unit of measure –Hours, defects, dollars, etc. Kim L. Jones CISM, CISSP, CRISC, MSIA GoodBad Inconsistent Results Expensive to Gather –Extremely Manual Highly Subjective –High/Medium/Low

6 “Contraxioms” Kim L. Jones CISM, CISSP, CRISC, MSIA

7 Contraxiom #1 -- Work Kim L. Jones CISM, CISSP, CRISC, MSIA For Geeks, Work is about solving problems Problems organize our thinking and provide a specific structure and approach Problem solving starts in the present. GeeksNon-Geeks For Non-Geeks, Work is about achieving a vision Visions are an imagined experience that get us out of bed in the morning. Vision realization starts in the future.

8 Contraxiom #1 -- Work Impact on Metrics –Do we truly understand the vision? And what the business must do/is trying to do to achieve that vision? –Are we relating our metrics TO the vision? This gives our metrics appropriate business context (the “So What?” factor) Kim L. Jones CISM, CISSP, CRISC, MSIA

9 Contraxiom #6 -- Lying Kim L. Jones CISM, CISSP, CRISC, MSIA For Geeks, Lying is evil. Truth is sacred. If you don’t know that it’s true, and you say it’s true, you’re lying. Exaggerations and opinions stated as fact are lies. GeeksNon-Geeks For Non-Geeks, Lying is not good. Lying is bad manners If you know that’s it’s false and say it’s true, you’re lying Exaggerations and opinions are part of normal speech.

10 Contraxiom #6 -- Lying Impact on Metrics –If exaggeration is normal speech, are our “metrics” accurate or exaggerated? Business can/will ask this…after all, “spin” is natural –When asked for specifics re: what will happen, are our qualifications of answers view as lack of commitment to our metrics/statements? Kim L. Jones CISM, CISSP, CRISC, MSIA

11 Asking the Right Question Is The Road Open? Kim L. Jones CISM, CISSP, CRISC, MSIA How close is the nearest rebel encampment? Are there mines on the road? What is the current state of rebel supplies? Is the destination still neutral

12 Asking the Right Question Are We Secure? Are We Compliant? What Is The Current Level of Risk? Are Our Controls Sufficient? Is The Risk Balanced Sufficiently To Achieve Our Vision? Kim L. Jones CISM, CISSP, CRISC, MSIA

13 Random Thoughts… Compliance Isn’t Always Bad Testing the Hypothesis Making the Subjective Objective Data Visualization Principles Kim L. Jones CISM, CISSP, CRISC, MSIA

14 Compliance Isn’t Always Bad Executives latch on to compliance because it meets the requirements of a good metric. The problem (as we all know) is that compliance doesn’t equal security –Worse, compliance does not equal appropriately balanced risk Even if you win the metrics battle, compliance will remain an issue if you are a regulated entity Possible (useful) workaround: measuring compliance with your policy framework –Meets compliance standards –Sets the risk floor! –Is in line with the vision! Kim L. Jones CISM, CISSP, CRISC, MSIA

15 Testing the Hypothesis… Gathering metrics to test hypothesis can be very useful when looking to ascertain and solve problems in your network. All previous rules re: metrics, context, etc. apply Remember: don’t prove the positive…disprove the negative. Kim L. Jones CISM, CISSP, CRISC, MSIA

16 Testing the Hypothesis… Corporate Mission: “Enable a Better Way for Trusted Commerce Infosec Mission: “We ensure the Trust in Trusted Commerce” –Trust defined as: your transactions will process as expected, when expected, how expected (i.e., without alteration). Hypothesis: Our Transactions Can be Trusted –Sub-Hypotheses: There are limited points of entry through which an outsider can get into our information systems Once inside, attackers cannot obtain access to internal systems because of strong passwords An intruder finding a hole somewhere cannot jump to core transactional systems Administrative credentials are difficult to obtain Kim L. Jones CISM, CISSP, CRISC, MSIA

19 Making the Subjective Objective… One of the complaints re: security metrics is an inconsistency in measurement –This undermines even the strongest/most significant metric as being opinion versus fact. Semi-qualitative metrics are a good starting point…but consider going a step further and implementing a standard evaluation checklist with relative values. Plotting the results of multiple assessments over a specific population may create a contextually relevant metric Kim L. Jones CISM, CISSP, CRISC, MSIA

20 Making the Subjective Objective Kim L. Jones CISM, CISSP, CRISC, MSIA

21 Data Visualization Principles 1.It’s All About The Data, Not the Design –Pretty designs and backgrounds are fun, but they exist to enhance the data, not overwhelm it 2.Simple Is Better –Erase what you don’t need –Avoid 3-D –Hint: Wizards aren’t necessarily helpful 3.Simplify the Color Palette –Muted, Primary Colors Kim L. Jones CISM, CISSP, CRISC, MSIA

22 Data Visualization Principles 4.Label Honestly and Accurately –Titles should be meaningful –Labels should enhance understanding –Always identify units of measure –Avoid clutter 5.Consider the Best Depiction of Data –Pie Chart? Stacked Bar? Pareto? 6.Test the Data! –Grant’s Captain Kim L. Jones CISM, CISSP, CRISC, MSIA

23 Wrapping it Up… Security is, at a fundamental level, a state of mind –Ditto for balanced risk It stands to reason, then, that measuring security and/or risk can be like catching a moonbeam –“What is the sound of one hand clapping?” Metrics and measurement are both art and science…you need to study both Make your metrics contextually relevant –What’s the vision? Be sure you’re answering the right question!! Kim L. Jones CISM, CISSP, CRISC, MSIA

Questions?

25 Contact Data… Kim L. Jones CISM, CISSP, CRISC, MSIA (480) 253-9120 Kljones.cism@gmail.com Kim L. Jones CISM, CISSP, CRISC, MSIA