1. Software Security Growth Modeling: Examining Vulnerabilities with Reliability Growth Models Andy Ozment Computer Security Group Computer Laboratory University of Cambridge First Workshop on Quality of Protection Milan, Italy September 15, 2005

2. Andy Ozment, University of Cambridge 2 Overview Reasons to measure software security Security growth modeling: using reliability growth models on a carefully collected data set Data collection process Data characterization challenges: failure vs. fault The problem of normalization Results of the analysis Future directions

3. Andy Ozment, University of Cambridge 3 Motivation Reduce the Market for Lemons effect –Info asymmetry in the market results in universally lower quality Security return on investment (ROI) –E.g. ROI for MS after it’s 2002 efforts Evaluate different software development methodologies Metrics needed for risk measurement and insurance We need a means of measuring software security –Ideal measure: $€£¥ –Goal: both absolute & relative measure

4. Andy Ozment, University of Cambridge 4 Security Growth Modeling Utilize software reliability growth modeling to consider security Problems –Data collection for faults is easier and more institutionalized –Hackers like abnormal data –Normalizing time data for effort, skill, etc. Previous work –Eric Rescorla: “Is finding security holes a good idea?” –Andy Ozment: “The Likelihood of Vulnerability Rediscovery and the Social Utility of Vulnerability Hunting” –5 th Workshop on Economics & Information Security (WEIS 2005)

5. Andy Ozment, University of Cambridge 5 Data Collection OpenBSD 2.2, December 1997 Vulnerabilities obtained from ICAT, Bugtraq, OSVDB, and ISS Search through source code –Identify ‘death date,’ when vulnerability was fixed –Identify ‘birth date,’ when vulnerability was first written Group vulnerabilities according to the version in which they were introduced

6. Andy Ozment, University of Cambridge 6 Data Characterization Problems Inclusion –Localizations –Specific hardware –Default install not vulnerable –Broad definition of vulnerability Uniqueness –Bundle patch from third-party –Simultaneous discovery of multiple related flaws Decided to try two perspectives –Failure: bundles & related were consolidated –Flaw: bundles & related were broken down into individual vulns

7. Andy Ozment, University of Cambridge 7 Data Normalization Normalize time data for effort, skill, holidays, etc. Not possible with this data This analysis of non-normalized data: ‘real-world security’ –Small business owner –Concerned with automated exploits An analysis of normalized data: ‘true security’ –Necessary for ROI, assessing development practices, etc. –Of concern to governments & high-value targets that may be the subject of custom attacks

8. Andy Ozment, University of Cambridge 8 Applying the Models Used SMERFS reliability modeling tool to test 7 models Analyzed both failure- and fault-perspective data sets –Failure data points:68 –Flaw data points:79 Models were tested for predictive accuracy –Bias (u-plots) –Trend (y-plots) –Noise No models were successful for flaw-perspective data Three models were successful for failure-perspective data. Most accurate successful model: Musa’s Logarithmic –Purification level (% of total vulns that have been found): 58.4% –After 54 months,the MTTF is: 42.5 days

9. Andy Ozment, University of Cambridge 9

10. 10 Future Research Normalize the data for relative numbers Examine the return on investment for a particular situation Utilize more sophisticated modeling techniques –E.g. recalibrating models Combine vulnerability analysis with traditional software metrics Compare this program with another

11. Andy Ozment, University of Cambridge 11 Conclusion Software engineers need a means of measuring software security Security growth modeling provides a useful measure However, the data collection process is time-consuming Furthermore, characterizing the data is difficult Nonetheless, the results shown here are encouraging More work is needed!

12. Andy Ozment, University of Cambridge 12 Questions? Andy Ozment Computer Security Group Computer Laboratory University of Cambridge

13. Andy Ozment, University of Cambridge 13 Number of vulnerabilities identified per year Perspective1998199920002001 ½ of 2002 Total Treated as failures1917 13268 Treated as flaws24182213279

14. Andy Ozment, University of Cambridge 14 Successful applicability results for models applied to the failure-perspective data: Statistic Musa’s Logarithmic Geometric Littlewood/Verrall Linear Prequential Likelihood 148.35(1)150.23(2)150.50(3) Bias (u-plot)0.12(1)0.13(2)0.18(3) Noise0.31(1)2.39(2)2.44(3) Trend (y-plot)0.20(3)0.18(2)0.14(3)

15. Andy Ozment, University of Cambridge 15 Estimates Made by Successful Models Statistic Musa’s Logarithmic Geometric Littlewood/Verrall Linear Initial Intensity Function 0.0590.0620.066 Current Intensity Function 0.0310.030 Purification Level0.5840.505N/A Current MTTF42.533.133.8