1. 12/9-10/2009 TGDC Meeting NIST-developed Test Suites David Flater National Institute of Standards and Technology http://vote.nist.gov

2. 12/9-10/2009 TGDC Meeting Context Status quo Labs have been testing to 2002, 1.0 (2005) Lots of proprietary, custom tooling and review processes August 2007 draft of 2.0 (1.9) NIST defined test methods where appropriate for entire scope In case of omission or error, fall back to status quo is an option May 2009 draft of 1.1 (1.01) Test methods for new and changed material back-ported from 2.0 (1.9) Status quo prevails for everything else Page 2 12/9-10/2009 TGDC Meeting

3. Materials Page 3 12/9-10/2009 TGDC Meeting 1.1 (1.01)2.0 (1.9) Usability & accessibility ✔✔ Security test suite ✔✔ Votetest ✔✔ Volume testing guidanceN/A ✔ Reliability, accuracy, … ✔ (In the draft) Hardware workmanshipOperating temperature and humidity ✔ EMCN/AComing soon Source code analyzer tool assessment guide and test suite …

4. 12/9-10/2009 TGDC Meeting General principles Ability to specify tests is limited by variability of design, unavailability of system-specific details, nature of requirements Avoid creating new requirements via test suite Measure when possible Measurable quantity Benchmark If not measurable, specify procedure for evaluating Test narratives Checklists Pass/fail criteria … Assessment by qualified experts Page 4 12/9-10/2009 TGDC Meeting

5. Coverage strategies The VVSG contains many different sorts of requirements… some more testable than others Coverage strategies vary "as appropriate" to what is being tested One test per requirement, one requirement per test At least one test per requirement, many requirements per test Many tests for one requirement Incidental testing Derived requirements Tested by exception Page 5 12/9-10/2009 TGDC Meeting

6. Usability & accessibility System-independent test narratives with pass/fail criteria CIF template & how-tos Common Industry Format for reporting usability test results Highly structured process surrounding the usability test(s) Voting Performance Protocol (VPP) is only in 2.0 Specified benchmarks Usability measured by test lab Page 6 12/9-10/2009 TGDC Meeting

7. Security test suite Procedures for evaluating Cryptography Access control Event logging Other security features Secure configurations If X is used, then it should be configured thusly… Not OEVT Page 7 12/9-10/2009 TGDC Meeting

8. Votetest Scope: basic, essential voting system logic Ability to define elections, capture, count, and report votes Voting variations Tests are intentionally simple… A volume test (mock election) is a significant test of all supported functions together …but they exercise the complete elections and voting process Election definition and test ballots in; reports out 92 tests formalized as SQL scripts Page 8 12/9-10/2009 TGDC Meeting

9. Volume testing guidance A volume test is specified in Part 3 of 2.0 Additional guidance for conducting that test has been published in a separate document Page 9 12/9-10/2009 TGDC Meeting

10. Reliability, accuracy, misfeed rate Improved test method replaces material that was historically included in the VSS/VVSG… hence, included in drafts Now evaluated using data collected during all tests, rather than a single, isolated test Page 10 12/9-10/2009 TGDC Meeting

11. Hardware workmanship General build quality Durability Maintainability Temperature and humidity Operating humidity requirement added to 1.1 Transportation and storage Page 11 12/9-10/2009 TGDC Meeting

12. Electromagnetic compatibility Coming soon for 2.0 Undergoing NIST internal review Page 12 12/9-10/2009 TGDC Meeting

13. One more thing Source code analyzer tool assessment guide and test suite Test labs use source code analyzers to help assess conformity to the VVSG Source code analyzers vary in quality This guide and test suite help to determine which are fit for purpose Page 13 12/9-10/2009 TGDC Meeting

14. Possible futures Voluntary use by test labs and/or manufacturers One input to a broader effort to reduce variability in testing Starting point for independent project(s) …manufacturer consortium? …jurisdictional testing? …general open source community? EAC requirements on test plans … not for NIST to determine Page 14 12/9-10/2009 TGDC Meeting