1. TGDC Meeting, December 2011 Michael Kass National Institute of Standards and Technology http://vote.nist.gov Update on SAMATE Automated Source Code Conformance Verification Against VVSG 1.0 Requirements

2. TGDC Meeting, December 2011 Outline Progress report on automated source code conformance verification effort introduced at last TGDC meeting Lessons learned Next steps Page 2

3. TGDC Meeting, December 2011 Software Assurance Metrics and Tool Evaluation (SAMATE) NIST/DHS co-sponsored project Goal - measure the effectiveness of software assurance tools to identify weaknesses and vulnerabilities in software Asked by EAC to assist voting system manufacturers and test labs improve a (heavily manual) source code review process by developing a uniform automated conformance verification capability for voting system source code against VVSG 1.0 software requirements Page 3

4. TGDC Meeting, December 2011 Tool Automation Progress Finished and tested automation of VVSG 1.0 rules for Java (still documenting C/C++ work) 50 custom rules written for the “Checkstyle” open source Java code analysis tool 41 verification tests (sample “bad” code) against those rules, to verify tool correctness against requirements Ran customized tool against 1M+ lines of Java source code Bundled Java tool rules and tests into ZIP and tar archive files for distribution to TGDC, VSTLs and manufacturers [1] "National Information Assurance Glossary"; CNSS Instruction No. 4009 National Information Assurance GlossaryNational Information Assurance Glossary Page 4

5. TGDC Meeting, December 2011 Requirement Automation Overview Identified 58 software requirements in VVSG (since last TGDC report) 26 requirements fully/partially automated for Java 12 requirements do not apply to Java language 14 requirements require entirely human analysis 3 requirements could be automated, but require “custom” knowledge about code or documentation 2 requirements could be verified with a “more capable” tool 1 requirement is ambiguous (not automatable) Page 5

6. TGDC Meeting, December 2011 Findings: Requirement Interpretation Issues We identified 10 requirement interpretation issues that require clarification The term “module” is often used in VVSG 1.0 without reference to type (method/function, class, library) What constitutes “intentional exception” is not defined Logical nesting limit requirement needs clearer definition Scoping clarification needed for names that differ by only one character Module line count requirement (type of module, and which lines “count”) not defined (we assumed module=method) Page 6

7. TGDC Meeting, December 2011 Findings: Requirement Interpretation Issues Additional requirement clarification needed for: Indentation requirements not specific (“clear” and “consistent” is all the requirement specifies) Scoping clarification needed for “unique” name comparison in code Functional, test and branch statements not clearly defined Definition of what a “library module” is for Java is not specified Lowest shared access for a resource is not clearly defined Page 7

8. TGDC Meeting, December 2011 Findings: Even “Clear” Requirements May Be Interpreted Differently An example: “Initializes every variable upon declaration where permitted” (VVSG 1.0 2:5.4.2) int varOne=0; // VVSG conformant static int varTwo; // non-VVSG-conformant What if Java automatically initializes varTwo to a value of 0 for you? In some cases, this happens at runtime Some manufacturers believe that the VVSG requirement is satisfied without explicitly assigning a value to a variable at declaration, and have written their code that way Page 8

9. TGDC Meeting, December 2011 Findings: Tool Automation Requires An Unambiguous Specification Need mutual understanding of VVSG requirement meaning among voting system manufacturers, VSTLS and the EAC The “devil is in the details” to unambiguously specify requirements for source code We observed some “disconnects” between voting system code and SAMATE interpretation of VVSG requirements (from our July 2011 visit to Wyle Labs) Page 9

10. TGDC Meeting, December 2011 Findings: Where Possible, SAMATE Should Customize Tools Manufacturers are Already Using Where voting system manufacturers are already using automated tools, SAMATE should (first) try to customize the tools that manufacturers are using Two manufacturers currently using “Checkstyle” tool for Java code conformance verification SAMATE rules can complement the Checkstyle rules the manufacturer already uses without “adding another tool” This requires a dialog between SAMATE and manufacturers to identify which tools they are using Page 10

11. TGDC Meeting, December 2011 Findings: Need to Maximize SAMATE Resources To make best use of our resources, SAMATE needs to know which tools, programming languages, compilers, libraries and coding conventions manufacturers are using 70% of voting system source code is written in C/C++, Java and C# languages (from discussion with VSTLs) Manufacturer’s compiler/library selection also dictate which tools SAMATE chooses for automation verification (e.g. GNU vs. Microsoft C++) If manufacturers are using industry coding conventions (not VVSG 1.0 requirements), then SAMATE should focus its automated verification efforts against those industry coding conventions Synchronize SAMATE work with future VSTL testing work Page 11

12. TGDC Meeting, December 2011 Next Steps TGDC review of customized Java tool and tests Discuss any issues with the SAMATE work Resolve those issues Distribute SAMATE tool rules and tests to manufacturers and VSTLs Two manufacturers are using the Checkstyle tool now (using their own custom rules) Get manufacturer/VSTL feedback on their interpretation and implementation of VVSG requirements in their tools Resolve those differences (via RFI or other means) Incorporate changes into future releases of VVSG as appropriate Publish a final distribution of customized tools and tests Page 12

13. TGDC Meeting, December 2011 Next Steps Have a discussion with voting system manufacturers and VSTLs to determine where SAMATE should focus its future efforts in: Tools Programming languages and compilers Coding conventions Future VSTL work (new systems, re-certifications) Page 13

14. TGDC Meeting, December 2011 Next Steps Investigate how SAMATE can contribute to automated conformance verification against VVSG 1.1 software integrity requirements Most VVSG 1.0 “style” requirements are removed in VVSG 1.1 (in lieu of using industry-accepted coding style conventions) VVSG 1.1 requirements are more “integrity focused” Not trivial to find integrity bugs in source code Requires more sophisticated tools than “style checkers” Requirements include identifying race conditions, deadlocks, livelocks, pointer validation, dynamic memory allocation, numeric overflows and CPU traps Page 14

15. TGDC Meeting, December 2011 Next Steps Integrate SAMATE VVSG work with SAMATE “Common Weakness Enumeration (CWE) Compatibility Effectiveness” program for testing automated vulnerability analysis tools CWE is a unified, measurable set of software weaknesses defined in an online dictionary sponsored by DHS and implemented by MITRE Tools are assessed against their ability to identify CWEs in software with known weaknesses and known source code “complexities” (that can influence a tool’s ability to correctly identify a weakness) SAMATE can leverage this work in developing test suites to verify tool effectiveness against VVSG source code integrity requirements Page 15

16. TGDC Meeting, December 2011 Summary SAMATE was successful at fully or partially automating Java source code verification against VVSG 1.0 requirements that could be automated Need to resolve semantic definitions of some VVSG 1.0 software requirements (with manufacturers, labs and EAC) for uniform automated conformance verification uniformly across all tools Need to prioritize SAMATE resources against voting system manufacturer’s tools, languages, compilers and coding conventions to maximize impact for future work Need to look forward to VVSG 1.1 software integrity requirements, and how SAMATE can assist manufacturers and VSTLs through tool effectiveness testing Page 16

17. TGDC Meeting, December 2011 Discussion Page 17