1. Measuring What Matters
Lisa Young
VP Cyber Risk Engineering
Axio Global

2. Data & Information

3. Terminology – Measure and Metric
A measure (or measurement) is the value of a specific characteristic of a given entity (collected data).
A metric is the aggregation of one or more measures to create a piece of business intelligence, in context.

4. Quiz - Measure or Metric?
I had 2 eggs for breakfast this morning.
It is 48 degrees Fahrenheit in Seattle today.
In our organization 3,000 staff have completed the required and updated security awareness training.
In our organization 3,000 staff out of 5,000 have completed the required security awareness training since it was updated in January By March 31, we are on track to ensure 98% of staff have completed security awareness training.

5. Why do you want to measure?

6. Getting started Not “What metrics should I use?” but
“What do I want to know or learn?”
Alternatives:
What decisions do I want to inform?
What actions do I want to take?
What behaviors do I want to change?

7. Why measure? Speak to decision-makers in their language
Demonstrate that the risk management or security program has measureable business value
Justify new investments; make improvements
Use trends to predict future events
Demonstrate that control objectives are (and continue to be) met
Answer key questions

8. Key questions When asked: How secure am I? Am I secure enough?
How much risk is acceptable?
What does this mean?
How secure am I compared to my competition?
Am I managing my risks well?
What is the business value of being more secure?
Of a specific security investment?
Do I need to spend more $$ on security or risk management? If so, on what?
What are the PR and legal impacts of a data breach?

9. Measurement objectives -1
Document the purposes for which measurement and analysis are done
Specify the kinds of actions that may be taken on the results of data analyses
May be identified at the operational unit level or the enterprise level
Sources can include
Monitoring of risk management process performance
Risk conditions
Compliance obligations
Industry benchmarks
Others?
Comment re overhead of measurement and analysis; need to be able to demonstrate resilience ROI
Dave White has an example about an organization we worked with that gave a project five years to improve resilience

10. Measurement objectives -2
May include
“Reduce the total number of controls under management”
“Maintain or improve supplier/customer performance against requirements”
“Improve uptime statistics”
“Improve risk identification”
“Software assets are kept up-to-date based on the criticality of the asset”
Once objectives are set, precise and quantifiable measures are established—can be base measure or derived
Example of base measure: Number of high-value assets by category
Example of derived metric: Percentage of high-value technology assets for which a risk assessment and analysis was conducted in last 12 months

11. So what? Why do you care? If I had this metric: (*)
What decisions would it inform?
What actions would I take based on it?
What behaviors would it affect?
What would improvement look like?
What would its value be in comparison to other metrics?
(*) informed by Douglas Hubbard, How to Measure Anything, John Wiley & Sons, 2010

12. Approach State a business objective
4/3/2019
Approach
State a business objective
Ideally your business objective supports a stated strategic objective
Ensure that [business unit, service, product, supply chain, technology, data center] is …
available to meet a specified customer or revenue growth objective
unavailable for no more than some stated period of time, number of transactions, other units of measure
fully compliant with [law, regulation, standard] so as not to incur [z] penalties

13. Who, what, where, when, why, how?
Who is the metric for? Who are the stakeholders? Who collects the measurement data?
What is being measured?
Where is the data/information stored?
When/how frequently are the metrics collected?
Why is the metric important (vs. others)?
The most meaningful information is conveyed by reporting trends over time vs. point in time metrics.
How is the data collected? How is the metric presented? How is the metric used?

14. To get started Identify sponsors and key stakeholders
Define measurement objectives and key questions
Determine information that informs these
What information do you already have?
What information do you need to collect?
What is the value of collecting additional information?
Define and vet a small number of key metrics
data collection
analysis procedures
number of metrics
number of participating business units
Collect, analyze, report, refine
Leverage an existing measurement program
Data visualization and compliance programs

15. Risk quantification
Building a risk quantification method or program is by definition “measuring” something.
There are foundational elements that need to be in place for a successful risk quantification program:
Business objectives and goals
Method and program
A set of questions that can be answered with the data; “clean” data
Process and workflow; roles and responsibilities
Results that are generated from data – minimizes “gaming” and provides context to compare results.
Governance and oversight of the method and program
We will not get to all of these topics in this workshop. I just wanted you to be thinking about the planning process that would get you to where you want to be.

16. Cost-effective vs. cost-benefit
Cost-benefit – for a given decision, one particular option has both a cost and a benefit.
This type of information may not be available on day one when building a measurement program.
Cost-effective – desired result or objective achieved by money spent.
Generally, this is a better representation of an information security and risk management program.
The information provided by the metrics will allow better decision-making. One of the questions to determine an effective security program is to ask: “is behavior changed by measuring X?” If the data is consistently ignored or not considered in decision-making, then the team may want to reconsider what data is being collected and why. (Wong, Security Metrics, pg.23)
Data collection and preparation for the analytics to be done are the bulk of the expenses for a measurement program. (Wong, pg.254)

17. Summary Good metrics are: those that are used often
answer important business and stakeholder questions
cost little to collect in relation to their value
are easily collected
do not require extensive manual intervention or manipulation.

18. Questions Lisa Young Vice President, Cyber Risk Engineering
Axio Global
LinkedIn:

19. GQIM process Objectives Goal Question Indicator Metric
Identify business objectives that establish the need for resilience and cybersecurity
Goal
Develop one or more goals for each objective
Question
Develop one or more questions that, when answered, help determine the extent to which the goal is met
These are the overall steps of the GQIM process that we will discuss in the rest of the workshop.
Indicator
Identify one or more pieces of information that are required to answer each question
Metric
Identify one or more metrics that will use selected indicators to answer the question