1. 1 INCOSE HRA Advanced Risk Management Conference 2007 Courtney Lane INCOSE HRA Risk Management Conference November 9, 2007 Its More Than Just Numbers: How to Implement a Qualitative Risk Analysis Approach This document is confidential and is intended solely for the use and information of the client to whom it is addressed.

2. 2 INCOSE HRA Advanced Risk Management Conference 2007 Purpose Describe the difference between qualitative and quantitative risk analysis Discuss common problems associated with qualitative analysis Examine best practices for implementing a qualitative risk analysis approach

3. 3 INCOSE HRA Advanced Risk Management Conference 2007 Standard Risk Management Process Develop Strategy This presentation will focus on the analysis sub-process. This sub-process should be defined in an organizations risk management strategy Analyze Plan Monitor and Control Identify

4. 4 INCOSE HRA Advanced Risk Management Conference 2007 Components of Risk Analysis Probability of occurrence Impact to a projects cost, schedule, and technical objectives Timeframe Handling method By identifying these components, project managers will be able to determine the magnitude of the risk and effectively prioritize handling of the risks

5. 5 INCOSE HRA Advanced Risk Management Conference 2007 Difference Between Qualitative and Quantitative Analysis Qualitative vs.Quantitative Use ranges to select probability and impact rankings Absolute numerical estimates are used for impact and probability Focuses on overall level of individual risks (e.g. high, medium, low) Allows projects to aggregate individual risks to determine the overall level of project risk Allows programs to quickly prioritize risks based on a number of criteria Uses complicated analysis such as Monte Carlo simulation to prioritize risks

6. 6 INCOSE HRA Advanced Risk Management Conference 2007 Why use Qualitative Analysis over Quantitative? Lack of historical data Lack of the necessary time and resources to conduct complicated analysis Desire to use criteria for prioritization that is difficult to quantify –Timeframe –Critical dependencies –Political considerations The type of analysis appropriate for an organization is dependent upon the risk management maturity level of the organization

7. 7 INCOSE HRA Advanced Risk Management Conference 2007 Risk Management Maturity Model LevelCharacteristics 1: Chaotic Risks are ignored or managed in an ad hoc fashion 2: Reactive Processes defined Management uncomfortable discussing risk Most efforts spent on resolving issues 3: Proactive Processes standardized across the organization Management comfortable with risk management Risk management data used in decision making processes 4: Predictive Use of quantitative analysis to predict future results Cost/benefit analysis implemented 5: Optimization Opportunities to exceed project objectives are identified and managed Source: Modified from SEIs Capability Maturity Model Most organizations are not at an appropriate maturity level to implement quantitative risk analysis

8. 8 INCOSE HRA Advanced Risk Management Conference 2007 Top Problems Related to Qualitative Analysis #1 Using inappropriate criteria to determine probability and consequence levels #2Trying to quantify qualitative data #3 Relying on risk scores generated by risk management tools to determine a list of top risks

9. 9 INCOSE HRA Advanced Risk Management Conference 2007 Problem #1: Inappropriate Criteria Criteria is often too detailed Subject matter experts lack the necessary expertise to appropriately select the criteria Easy to game the analysis since justification of criteria selection is extremely subjective Poor criteria definitions make it difficult for risk analysts to accurately reflect impacts to cost, schedule, and performance

10. 10 INCOSE HRA Advanced Risk Management Conference 2007 Solution: Understand Your Audience Determine what questions decision makers want the risk management process to answer Assess the level of expertise of those performing the risk assessments Select criteria that balances the needs and abilities of both parties

11. 11 INCOSE HRA Advanced Risk Management Conference 2007 Examples of Qualitative Analysis Criteria Negligible (A) Marginal (B) Significant (C) Critical (D) Catastrophic (E) Cost Minimal or no impact $25K-$50K$50K-100K$100K-250K>$250K Schedule Schedule slip < 7 days Schedule slip between 7 and 14 days Schedule slip between 14 and 21 days Schedule slip between 21 and 30 days Schedule slip greater than 1 month Performance Minimal or no impact No impact to key performance parameter; degraded capability performance Degraded enterprise capability performance Degraded capability performance with impacts to one key performance parameter Degraded capability performance with impacts to more than one key performance parameter Criteria is too detailed and thus difficult to assess

12. 12 INCOSE HRA Advanced Risk Management Conference 2007 Examples of Qualitative Analysis Criteria Negligible (A) Marginal (B) Significant (C) Critical (D) Catastrophic (E) Cost Minimal or no impact <5% of overall budget 5-7%of overall budget 7-10% of overall budget >10% of overall budget Schedule Minimal or no impact Additional resources required; able to meet need dates Minor slip in key milestones; not able to meet need dates Major slip in key milestone or critical path impacted Cant achieve key team or major project milestone Performance Minimal or no impact Acceptable with some reduction in margin Acceptable with significant reduction in margin Acceptable; no remaining margin Unacceptable Source: Risk Management Guide for DoD Acquisition Criteria is extremely subjective and could be easily gamed

13. 13 INCOSE HRA Advanced Risk Management Conference 2007 Examples of Qualitative Analysis Criteria Negligible (A) Marginal (B) Significant (C) Critical (D) Catastrophic (E) Cost Minimal or no impact Cost impacts contained within the project Cost impacts contained within the program Cost impacts contained within the organization Cost impacts affect external partners and could jeopardize program Schedule Minimal or no impact Impacts to critical path contained within the project Impacts to critical path contained within the program Impacts to critical path of more than one program in the organization Impacts the critical path of external partners and could jeopardize program Performance Minimal or no impact Impacts to critical functions of the project Impacts to critical functions of the program Impacts to critical functions of the organization Impacts to critical functions of external partners and could jeopardize the program Criteria definitions are easy to assess and impact levels are consistent across criteria

14. 14 INCOSE HRA Advanced Risk Management Conference 2007 Problem #2: Quantifying Qualitative Data Engineers often feel more comfortable using numbers when performing analysis –Assign numbers to levels of impact –Use specific probabilities instead of ranges –Risk scores often determined by multiplying impact by probability However, most risk analysis uses uncalibrated ordinal scales –Cannot perform math on uncalibrated scales –Gives a false sense of rigor –Decisions made on inaccurate data Performing math on qualitative data may lead to inaccurate conclusions and a false sense of rigor

15. 15 INCOSE HRA Advanced Risk Management Conference 2007 Solution: Do Not Include Numbers in Analysis Avoid assigning numbers to levels of impact and probability –Use probability ranges –Use letters to represent levels of impact and probability Use the highest level of individual impacts to determine overall impact (dont average impacts) Use a risk matrix to determine overall level of risk

16. 16 INCOSE HRA Advanced Risk Management Conference 2007 Example Qualitative Risk Analysis Overall level of risk is Red

17. 17 INCOSE HRA Advanced Risk Management Conference 2007 Problem #3: Overemphasizing Risk Scores If problem #2 is not overcome, this becomes a much larger problem Even if numerical scores are not used, rankings could be incorrect –Risk tools can generate a list of top risks based on locations on the risk matrix –Cannot rely solely on the placement on the risk matrix –Dependencies, political issues and timeliness factors also play a part Overemphasis of risk scores to determine top risks results may result in poor decisions made by senior leaders

18. 18 INCOSE HRA Advanced Risk Management Conference 2007 Solution: Tailor list of top risks based on the audience Consider all factors, not just likelihood and consequence, when creating the list Do not use arbitrary thresholds to determine handling strategies Aim to manage all yellow and red risks

19. 19 INCOSE HRA Advanced Risk Management Conference 2007 Summary and Conclusion Select the appropriate type of analysis based on the risk management maturity level of the organization Typical challenges associated with qualitative analysis can be overcome: –Know your audiences –Avoid using numbers to perform analyses –Consider other factors when selecting list of top risks Qualitative analysis can provide leaders with enough information to make sound decisions if implemented properly

20. 20 INCOSE HRA Advanced Risk Management Conference 2007 Contact Information Courtney Lane Senior Consultant Booz Allen Hamilton lane_courtney@bah.com 703-961-8009