1. Copyright © 2004 - The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License. The OWASP Foundation OWASP AppSec June 2004 NYC http://www.owasp.org The Need for Metrics and Measurement in Application Security Jack Danahy OWASP Metrics and Measurement Standards Committee Project Lead jack.danahy@ouncelabs.com 781-290-5333

2. OWASP AppSec 2004 2 The Need for Metrics Identify critical areas of focus Set security investment priorities Track effectiveness of remediation and training Target critical remediation needs Evaluate ROI in security training investment Set and monitor security acceptance criteria Monitor compliance with established thresholds Publish trend analyses to document security efforts/progress Evaluate outsourcers’ compliance with contractual requirements Identify critical vulnerabilities early Learn how to fix the vulnerability Confirm vulnerability elimination Monitor performance of development teams and outsourcers Set critical priorities and security exit criteria Publish results Prexis Vulnerability Analysis Data Compliance/ Audit Managers Developers Program Managers Development Managers CSO/CISO

3. OWASP AppSec 2004 3 OWASP Metrics and Measurement Project Goals  Member survey and outreach to characterize significant and required metrics  Metrics gathering best practices framework  Recommendations for metrics gathering, tool analysis, metrics aggregation and weighting

4. OWASP AppSec 2004 4 The Case for Measurement The Need for Metrics:  Certification  Prioritization  Remediation  Tracking

5. OWASP AppSec 2004 5 Metrics for Certification  Governance  Credible, reliable metrics support compliance efforts by demonstrating pervasive security  Stability  Proof of security and lack of excessive patching increase customer confidence and reduce operational risk  Functionality  Validation of appropriate implementation of defined security components ensures that product meets baseline security requirements

6. OWASP AppSec 2004 6 Metrics for Prioritization  Determine application or project vulnerability  Determine severity of vulnerabilities  Prioritize remediation efforts low exposure Audience and Exposure high exposure Low Value High

7. OWASP AppSec 2004 7 Metrics for Remediation  Informed business-level decision support  Legacy applications: Wrap it, rewrite it, or replace it  Outsourced projects: Baselines and thresholds drive acceptance criteria and accountability  Resource allocation: focus investments and attention  Efficient workflow for developers  Specific identification of vulnerability  Explanation of vulnerability including potential impact  Conclusive remediation recommendations

8. OWASP AppSec 2004 8 Metrics for Tracking  Establish baseline and acceptable thresholds  Set accountability expectations with external vendors  Measure team performance  Provide reliable information to all areas of organization  Monitor progress over time requires:  Granularity of information  Periodicity of data (regulatory and public company requirements)

9. OWASP AppSec 2004 9 Sample Outsourcer Report Card

10. OWASP AppSec 2004 10 The Case for Measurement  Certification: Provide quantifiable measurement of security  Prioritization: Make informed resource allocation decisions  Remediation: Identify and eliminate risks caused by vulnerabilities  Tracking: Prove progress against reliable baselines and thresholds

11. OWASP AppSec 2004 11 Call for Participation  Active recruitment efforts underway  owasp-metrics@lists.sourceforge.net  Questions? Comments?  Contact me at: jack.danahy@ouncelabs.com

12. OWASP AppSec 2004 12 Thank you