TGDC Meeting, July 2011 VVSG 1.1 Test Suite Status Mary Brady Manager, NIST Information Systems Group, Software and Systems Division, ITL

TGDC Meeting, July 2011Page 2 Background Status quo: Labs have been testing to VVSG 1.0 (2005) using proprietary, custom tooling and review processes In , NIST developed a set of public test suites for VVSG 2.0, to be used as part of the EAC Testing and Certification Program In 2009, to support VVSG 1.1, test methods for new and changed material was back-ported from 2.0 test suites; status quo prevails for everything else

TGDC Meeting, July 2011Page 3 Test Validation NIST test suites have been rigorously traced to VVSG requirements Reviewed by independent parties, including the EAC, VSTL labs, experts, and the public Procedural validation has been on-going Operating temperature and humidity -- complete Usability and accessibility – ongoing Core functionality, security under consideration Pursued obtaining in-house voting systems to aid in the evaluation of NIST test suites

TGDC Meeting, July 2011Page 4 Recent Activities NIST met with each accredited VSTL to discuss possible integration of core functionality tests Recent activities regarding changes to VVSG 1.1 will require changes to the NIST developed test suites; delayed status of VVSG 2.0 warrants additional focus on improving testing for VVSG 1.1, including existing methods for VVSG 1.0 NIST and EAC met with both accredited VSTL’s to discuss ways to improve testing for the evolving VVSG 1.1 and to integrate NIST-developed tests into the certification process

TGDC Meeting, July 2011Page 5 Preliminary Findings To ensure consistency across testing laboratories, agreed upon test methods must be developed for all of VVSG 1.1 Face-to-face meetings revealed differences in interpretation of the term “readiness test”. Labs read as “acceptance test” whereas NIST definition was a “simple health check” Some NIST tests require new election definitions, which are problematic for the labs Labs define test data separately from the test case; NIST defines them together

TGDC Meeting, July 2011Page 6 Additional Findings Some NIST tests should be reworked to take into account practical test methods, with temperature and humidity cited as an example Source code analysis represents a large percentage of the cost of testing; tools remain difficult to use Reference ballots, including guidance on how ballots should be marked (pre-marked, hand-marked), election definitions/voting patterns to be used, and complexity of precincts/ballot styles, would be useful Should identify priority areas outside of VVSG 1.1

TGDC Meeting, July 2011Page 7 Action Plan Monthly meetings among EAC, NIST, VSTL’s, with bi- annual face-to-face meetings: Focus on developing common test methods for new VVSG 1.1, starting with information from EAC matrix, NIST test suites, and existing VSTL definitions Validate NIST test suites and integrate into certification process Expand effort on source code analysis tools and work with the VSTL’s to develop rules for VVSG 1.1 Explore creating a set of reference ballots to be used by all labs

TGDC Meeting, July 2011 Discussion/Questions Page 8