Security Attribute Evaluation Method: A Cost Benefit Analysis Shawn A. Butler Computer Science Department Carnegie Mellon University 9 November 2001

M S Hey Boss, we need more security. I think we should get the new Acme 2000 Hacker Abolisher We always seem to need more security! Don’t we have enough?

M S Trust me, we will be more secure! What are my alternatives? What is it going to cost? What is the added value?

Value? Alternatives? S

Problem Security managers lack structured cost- benefit methods to evaluate and compare alternative security solutions.

Security Architecture Development Process Risk Assessment Outcomes Threats Prioritized Risks Select Countermeasures System Design Policies Requirements Available Countermeasures Security Components Develop Security Architecture

Security Architecture Development Process Risk Assessment Outcomes Threats Prioritized Risks Select Countermeasures System Design Policies Requirements Available Countermeasures Security Components Develop Security Architecture

The Multi Attribute Risk Assessment 1.Determine threats and outcomes 2.Assess outcome attribute values 3.Assess weights 4.Compute threat indices 5.Sensitivity Analysis Risk Assessment Outcomes Threats Prioritized Risks

Threats Scanning Procedural Violation Browsing Distributed Denial of Service Password Nabbing Personal Abuse Signal Interception : 29 Threats Determine Threats and Outcomes Outcome Attributes Lost Productivity Lost Revenue Regulatory Penalties Reputation Lives Lost Lawsuits : O i = (Lost Prod, Lost Rev, Reg Penalties, Reputation)

Assess Outcome Attribute Values Outcomes Attacks Lost Producti- vity (hrs) Lost Revenue ($$) Regulatory Penalties (scale 0-6) Reputation (scale 0-6) Scanning 10,220/yr (3-4/hr) Low.3001 Expected.5201 High 11,00004 Procedural Violation 4,380/yr (1-2/hr) Low 0000 Expected 2201 High 4012,00034

Prioritize and Assess Weights (Swing Weight Method) Best Worst Lost Prod Lost Rev Reg Penal Reputation 240 hrs $12, hrs $0 0 Rank Weight (w i ) Order

Compute Threat Indices Hours + $$ + Reputation + Regulatory Penalties = ? 1 0 P: Lost Productivity R: Reputation G: Regulatory Penalties L: Lost Revenue , So determine Value Functions V j (x j ) L(x 1 ) $$ + P(x 2 )Hours + R(x 3 )Reputation + G(x 4 )Regulatory Penalties = TI Nonsense !

Computing the Threat Index p expected  (  j=attributes W j  V j (x j expected )) Expected threat TI a = Freq a  [ p low  (  j=attributes W j  V j (x j low )) + p expected  (  j=attributes W j  V j (x j expected )) + p high  (  j=attributes W j  V j (x j high )) ] Threat index

Scanning in More Detail Outcomes Attacks Lost Producti- vity (hrs) Lost Revenue ($$) Regulatory Penalties (scale 0-6) Reputation (scale 0-6) Scanning 10,220/yr Low.3001 Expected.5201 High 11, = p low  (  j=attributes W j  V j (x j low )).07 = p expected  (  j=attributes W j  V j (x j expected )).00 = p high  (  j=attributes W j  V j (x j high )) 10,220  ( ) 

Risk Assessment Results Threat FrequencyLowExpectedHighTotal Scanning 10, Procedural Violation Browsing Dist Denial of Service Password Nabbing Personal Abuse TOTAL 1,507.18

But what about the numbers?

Risk Assessment Sensitivity Analysis Attack Frequencies Outcome Attribute Values Attribute Weights

Probability Distributions Scanning Frequency Dist Scanning Reputation Dist

Change in TI Rankings ?

Cryptographic Compromise Distribution

Regression Sensitivity.078 Lost Productivity/K30.19 Reputation/w j Reputation Outcome

Sensitivity Analysis How sensitive are the answers to estimation errors? Does it matter if the estimates are not accurate? How accurate do they have to be before the decision changes? When is it important to gather additional information?

Selecting Countermeasures Risk Assessment Outcomes Threats Prioritized Risks Select Countermeasures System Design Policies Requirements Available Countermeasures Security Components Develop Security Architecture

Security Attribute Evaluation Method (SAEM) What is SAEM? A structured cost-benefit analysis technique for evaluating and selecting alternative security designs Why SAEM? Security managers make explicit their assumptions Decision rationale is captured Sensitivity analysis shows how assumptions affect design decisions Design decisions are re-evaluated consistently when assumptions change Stakeholders see whether their investment is consistent with risk expectations

SAEM Process Evaluation Method 1.Assess security technology benefits 2.Evaluate security technology benefits 3.Assess coverage 4.Analyze Costs Select Countermeasures System Design PoliciesRequirements Available Countermeasures Security Components Prioritized Risks

Assess Security Technology Benefits Scanning50%75%66% 33% 50% Procedural Violation 50%40%25% Browsing30% Dist Denial of Service 75% Password Nabbing 50% Personal Abuse40% Effectiveness Percentages Threat Security Tech PF FirewallPrxy Firewall Net IDSAuditing Host IDS Vuln Assess Hardened OS Auth Policy Serv Virtual Priv Net Net Monitors

Evaluate Security Technology Benefits Scanning (886) Procedural Violation (367) Browsing (226) 158 Dist Denial of Service (26.12) 6.6 Password Nabbing (.62).31 Personal Abuse (.13).08 Threat Security Tech PF FirewallPrxy FirewallNet IDSAuditingHost IDSVuln AssessHardened OS Auth Policy Serv Virtual Priv NetNet Monitors

Prioritized Technologies Technology  Value Threat Index Overall Rank PKI/Cert.2428 Auditing Auth Policy Server Host-IDS 5892 Net-IDS Smart Cards One Time Psswrd 3407 Single Sign-on 035

Assess Coverage

Host Intrusion Detection Coverage

Auditing Coverage

Analyze Costs $0  Host IDS  Single Sign-on  Smart Cards  Net IDS  Auditing  PKI Cert $20,000  Auth Policy Server Threat Index  Purchase Cost

SAEM Sensitivity Analysis The vulnerability Assessment tool is 66% effective. What does that really mean?

Security Technology Effects on the Risk Assessment Benefit Estimates: - Reduce Frequency - Change Outcomes Vulnerability Assess Scanner Benefit Distribution

Top 25 Countermeasure Rankings Reduced Frequency

Countermeasure Rank Overlaps

Outcome Changes Procedural Violations Reputation Before After

Preliminary Results Risk Assessment threat indices reflect security manager’s concerns –based on interviews and feedback Security managers are able to estimate technology benefits –based on experience, organizational skill levels, and threat expectations Sensitivity Analysis is key to method –based on uncertainty of assumptions