1. Security Attribute Evaluation Method: A Cost Benefit Analysis Shawn A. Butler Computer Science Department Carnegie Mellon University 9 November 2001

2. M S Hey Boss, we need more security. I think we should get the new Acme 2000 Hacker Abolisher We always seem to need more security! Don’t we have enough?

3. M S Trust me, we will be more secure! What are my alternatives? What is it going to cost? What is the added value?

4. Value? Alternatives? S

5. Problem Security managers lack structured cost- benefit methods to evaluate and compare alternative security solutions.

6. Security Architecture Development Process Risk Assessment Outcomes Threats Prioritized Risks Select Countermeasures System Design Policies Requirements Available Countermeasures Security Components Develop Security Architecture

7. Security Architecture Development Process Risk Assessment Outcomes Threats Prioritized Risks Select Countermeasures System Design Policies Requirements Available Countermeasures Security Components Develop Security Architecture

8. The Multi Attribute Risk Assessment 1.Determine threats and outcomes 2.Assess outcome attribute values 3.Assess weights 4.Compute threat indices 5.Sensitivity Analysis Risk Assessment Outcomes Threats Prioritized Risks

9. Threats Scanning Procedural Violation Browsing Distributed Denial of Service Password Nabbing Personal Abuse Signal Interception : 29 Threats Determine Threats and Outcomes Outcome Attributes Lost Productivity Lost Revenue Regulatory Penalties Reputation Lives Lost Lawsuits : O i = (Lost Prod, Lost Rev, Reg Penalties, Reputation)

10. Assess Outcome Attribute Values Outcomes Attacks Lost Producti- vity (hrs) Lost Revenue ($$) Regulatory Penalties (scale 0-6) Reputation (scale 0-6) Scanning 10,220/yr (3-4/hr) Low.3001 Expected.5201 High 11,00004 Procedural Violation 4,380/yr (1-2/hr) Low 0000 Expected 2201 High 4012,00034

11. Prioritize and Assess Weights (Swing Weight Method) Best Worst Lost Prod Lost Rev Reg Penal Reputation 240 hrs $12,000 3 4 0 hrs $0 0 Rank 100 20 40 80 Weight (w i ).42.08.17.33 Order 1 2 4 3

12. Compute Threat Indices Hours + $$ + Reputation + Regulatory Penalties = ? 1 0 P: Lost Productivity R: Reputation G: Regulatory Penalties L: Lost Revenue 1 0 1 0 1 0 12,000 240 3 4 0 0 0 0 So determine Value Functions V j (x j ) L(x 1 ) $$ + P(x 2 )Hours + R(x 3 )Reputation + G(x 4 )Regulatory Penalties = TI Nonsense !

13. Computing the Threat Index p expected  (  j=attributes W j  V j (x j expected )) Expected threat TI a = Freq a  [ p low  (  j=attributes W j  V j (x j low )) + p expected  (  j=attributes W j  V j (x j expected )) + p high  (  j=attributes W j  V j (x j high )) ] Threat index

14. Scanning in More Detail Outcomes Attacks Lost Producti- vity (hrs) Lost Revenue ($$) Regulatory Penalties (scale 0-6) Reputation (scale 0-6) Scanning 10,220/yr Low.3001 Expected.5201 High 11,00004.01 = p low  (  j=attributes W j  V j (x j low )).07 = p expected  (  j=attributes W j  V j (x j expected )).00 = p high  (  j=attributes W j  V j (x j high )) 10,220  (.01 +.07 +.00)  886.57

15. Risk Assessment Results Threat FrequencyLowExpectedHighTotal Scanning 10,220.0084.0750.0034886.57 Procedural Violation 4380.0000.0773.0065367.03 Browsing 2920.0000.0742.0035226.71 Dist Denial of Service 156.0085.1530.006026.12 Password Nabbing 365.0001.0008.0009.62 Personal Abuse 110.0000.0003.0009.13 TOTAL 1,507.18

16. But what about the numbers?

17. Risk Assessment Sensitivity Analysis Attack Frequencies Outcome Attribute Values Attribute Weights

18. Probability Distributions Scanning Frequency Dist Scanning Reputation Dist

20. Change in TI Rankings ?

21. Cryptographic Compromise Distribution

22. Regression Sensitivity.078 Lost Productivity/K30.19 Reputation/w j -.213 Reputation Outcome-.639 -0.75-0.5-0.2500.250.50.751

23. Sensitivity Analysis How sensitive are the answers to estimation errors? Does it matter if the estimates are not accurate? How accurate do they have to be before the decision changes? When is it important to gather additional information?

24. Selecting Countermeasures Risk Assessment Outcomes Threats Prioritized Risks Select Countermeasures System Design Policies Requirements Available Countermeasures Security Components Develop Security Architecture

25. Security Attribute Evaluation Method (SAEM) What is SAEM? A structured cost-benefit analysis technique for evaluating and selecting alternative security designs Why SAEM? Security managers make explicit their assumptions Decision rationale is captured Sensitivity analysis shows how assumptions affect design decisions Design decisions are re-evaluated consistently when assumptions change Stakeholders see whether their investment is consistent with risk expectations

26. SAEM Process Evaluation Method 1.Assess security technology benefits 2.Evaluate security technology benefits 3.Assess coverage 4.Analyze Costs Select Countermeasures System Design PoliciesRequirements Available Countermeasures Security Components Prioritized Risks

27. Assess Security Technology Benefits Scanning50%75%66% 33% 50% Procedural Violation 50%40%25% Browsing30% Dist Denial of Service 75% Password Nabbing 50% Personal Abuse40% Effectiveness Percentages Threat Security Tech PF FirewallPrxy Firewall Net IDSAuditing Host IDS Vuln Assess Hardened OS Auth Policy Serv Virtual Priv Net Net Monitors

28. Evaluate Security Technology Benefits Scanning (886) 443223301 594 443 Procedural Violation (367) 183220274 Browsing (226) 158 Dist Denial of Service (26.12) 6.6 Password Nabbing (.62).31 Personal Abuse (.13).08 Threat Security Tech PF FirewallPrxy FirewallNet IDSAuditingHost IDSVuln AssessHardened OS Auth Policy Serv Virtual Priv NetNet Monitors

29. Prioritized Technologies Technology  Value Threat Index Overall Rank PKI/Cert.2428 Auditing 24111 Auth Policy Server 16115 Host-IDS 5892 Net-IDS 29310 Smart Cards 10316 One Time Psswrd 3407 Single Sign-on 035

30. Assess Coverage

31. Host Intrusion Detection Coverage

32. Auditing Coverage

33. Analyze Costs 0 589 $0  Host IDS  Single Sign-on  Smart Cards  Net IDS  Auditing  PKI Cert $20,000  Auth Policy Server Threat Index  Purchase Cost

34. SAEM Sensitivity Analysis The vulnerability Assessment tool is 66% effective. What does that really mean?

35. Security Technology Effects on the Risk Assessment Benefit Estimates: - Reduce Frequency - Change Outcomes Vulnerability Assess Scanner Benefit Distribution

36. Top 25 Countermeasure Rankings Reduced Frequency

37. Countermeasure Rank Overlaps

38. Outcome Changes Procedural Violations Reputation Before After

39. Preliminary Results Risk Assessment threat indices reflect security manager’s concerns –based on interviews and feedback Security managers are able to estimate technology benefits –based on experience, organizational skill levels, and threat expectations Sensitivity Analysis is key to method –based on uncertainty of assumptions