1. Software Assurance Metrics and Tool Evaluation (SAMATE) Michael Kass National Institute of Standards and Technology http://samate.nist.gov/ Michael.kass@nist.gov

2. June 8, 2005 2 Outline Overview of Software Assurance (SwA) tool testing at NIST Overview of Software Assurance (SwA) tool testing at NIST Description of SAMATE project Description of SAMATE project Follow-on Follow-on

3. June 8, 2005 3 Dept Homeland Security Concern Do software assurance tools work as they should? Do software assurance tools work as they should? Do they really find vulnerabilities and catch bugs? How much assurance does running the tool provide? Do they really find vulnerabilities and catch bugs? How much assurance does running the tool provide? Software Assurance tools should be: Software Assurance tools should be: Tested (accurate and reliable) Tested (accurate and reliable) Peer reviewed Peer reviewed Generally accepted Generally accepted

4. June 8, 2005 4 Goals of SAMATE Develop metrics for the effectiveness of SwA tools and to identify deficiencies in software assurance methods and tools Develop metrics for the effectiveness of SwA tools and to identify deficiencies in software assurance methods and tools Perform SwA R&D to assess current methods and tools in order to identify deficiencies which can lead to software product failures and vulnerabilities Perform SwA R&D to assess current methods and tools in order to identify deficiencies which can lead to software product failures and vulnerabilities Identify gaps in methods and tools and suggest areas of research Identify gaps in methods and tools and suggest areas of research

5. June 8, 2005 5 The NIST SAMATE Project (Software Assurance Metrics and Tool Evaluation) 1. Conduct surveys Tools Tools Researchers and companies Researchers and companies 2. Host workshops & conference sessions Taxonomy of SwA functions and techniques Taxonomy of SwA functions and techniques Order of importance (cost/benefit, criticalities, …) Order of importance (cost/benefit, criticalities, …) Gaps and research agendas Gaps and research agendas Studies to develop tool effectiveness metrics Studies to develop tool effectiveness metrics 3. Evaluate tools Detailed specification Detailed specification Test plans Test plans Host reference dataset library Host reference dataset library

6. June 8, 2005 6 A Taxonomy of Static Analysis Tool Functions Language Language Source/Binary analysis Source/Binary analysis Semantic checking (abstract syntax tree) Semantic checking (abstract syntax tree) Interprocedural analysis Interprocedural analysis Strong type checking (type casting vulnerabilities, uninitialized variable use) Strong type checking (type casting vulnerabilities, uninitialized variable use) Memory allocation checking (memory leaks, deallocation of unallocated memory) Memory allocation checking (memory leaks, deallocation of unallocated memory) Logic checking (unnecessary code, unreachable code) Logic checking (unnecessary code, unreachable code) Interface checking (include file cycling) Interface checking (include file cycling) Security checking Security checking Buffer overflow/underflow Buffer overflow/underflow Stack overflows Stack overflows Heap overflows Heap overflows Integer overflow/underflow Integer overflow/underflow Tainted data Tainted data Error path problems Error path problems Locking problems Locking problems Code metric generation (LOC, number of methods, levels of inheritance) Code metric generation (LOC, number of methods, levels of inheritance)

7. June 8, 2005 7 SA Tool Effectiveness Metrics What constitutes a tool’s effectiveness metric? Number of defects detected vs. total defects Number of defects detected vs. total defects Number of false positives Number of false positives Number of false negatives Number of false negatives …

8. June 8, 2005 8 Documenting tool effectiveness Tool functional specification Tool functional specification Test plan Test plan Reference dataset Reference dataset Test report Test report

9. June 8, 2005 9 T (mos.) 126 121824 Workshop1 SA classes 3459 1521 Workshop 3 Define Metric Workshop 2 fill gaps focus group class 1 focus group class 1 Function Taxonomy Tool Survey Publication select func strawman spec test plan draft Spec0 Spec1 test plan select func strawman spec draft Spec0 Spec1 SAMATE Project Timeline focus group class 2 focus group class 2 tool testing matrix test reports

10. June 8, 2005 10 Contact for SAMATE Participation Paul Black Paul Black Project Leader, Software Diagnostics & Conformance Testing Division, Software Quality Group Project Leader, Software Diagnostics & Conformance Testing Division, Software Quality Group paul.black@nist.gov paul.black@nist.gov