1. Software Diagnostics and Conformance Testing CRT: Voting system logic testing (Votetest) David Flater 2008-01-23

2. Software Diagnostics and Conformance Testing Players and field Congress –Help America Vote Act (HAVA) –National Voting Rights Act –Section 508 –Americans with Disabilities Act Election Assistance Commission (EAC) –Voluntary Voting System Guidelines (VVSG) –Manufacturer registration, lab accreditation, certification, … Technical Guidelines Development Committee (TGDC) NIST –National Voluntary Lab Accreditation Program (NVLAP) Voting system manufacturers Voting jurisdictions State and local election officials Concerned citizens Professional advocates Academics Reporters & bloggers (and they all have lawyers)

3. Software Diagnostics and Conformance Testing Manufacturer-driven activities Conformity assessment –Physical configuration audit –Documentation and design reviews –Electromagnetic compatibility and environmental testing –Logic testing (Votetest) –Volume test (mock election) –CRT benchmarks –STS and HFP testing Election Assistance Commission (EAC) certification Jurisdiction acceptance testing and certification Deployment Monitoring Logic testing in context

4. Software Diagnostics and Conformance Testing Goals Status quo: test labs are on their own to develop conformance tests for the Voluntary Voting System Guidelines (VVSG) Conservative goal: reduce variability and cost of testing by providing test labs with tools and materials useful in constructing test suites Ambitious goal: further reduce variability and cost by providing a canonical test suite

5. Software Diagnostics and Conformance Testing Choosing the right tools “Testing target:” the object of conformity assessment –A.k.a. Implementation/Device/System Under Test Different kinds of testing targets need different testing approaches

6. Software Diagnostics and Conformance Testing Differences from other testing targets Automatic testing is not feasible –Don’t have standard interfaces to get data in and results out –Voters are part of the process (people in the loop) –Unanticipated nonfatal errors must be detected Cost of executing tests is a major issue –Significant time and effort to prepare election definitions, ballot styles, and test ballots or voters for each test case –Labor costs for people in the loop –Politics: any increase in total cost for certification will be considered an unfunded mandate More is not better –A vote is a vote (logically) –As we increase the number of votes counted Cost of testing increases proportionally Return on investment diminishes rapidly Context: one step in a long process –Volume test (mock election), logic verification, etc.

7. Software Diagnostics and Conformance Testing The requirements Normative reference: the next iteration of the VVSG (in public review) Logic must correctly handle all voting variations that the manufacturer claims to support Everything must work through the complete elections and voting process Election definition Ballot definition Configuration and calibration of equipment Logic and accuracy testing Vote gathering TabulationReconciliationReporting 1 of M votingXXXXXXXX N of M votingXXXXXXXX Cumulative votingXXXXXXXX Ranked order votingXXXXXXXX In-person votingXXXXXXXX Absentee votingXXXXXXXX Provisional / challenged ballots XXXXXXXX Write-insXXXXXXXX Review-required ballots XXXXXXXX Primary electionsXXXXXXXX Split precinctsXXXXXXXX Ballot rotationXXXXXXXX Straight party votingXXXXXXXX Cross-party endorsement XXXXXXXX

8. Software Diagnostics and Conformance Testing Testing strategy All tests are end-to-end tests that exercise the complete elections and voting process Small number (10-100) of carefully selected tests –Cover each voting variation with a simple, synthetic test (around 10 ballots, 1 contest) –Similarly cover all meaningful pairs of voting variations –Few slightly larger tests (around 100 ballots, multiple contests) based on real sample ballots –Few miscellaneous tests (e.g., boundary cases) Test scripts to be “realized” according to the specifics of the target Test oracle No big tests in this test suite –Context: The big volume test (mock election) provides a significant test of all supported voting variations together Punt devilish details –Some requirements are too implementation-dependent –Some requirements are incidental to every scenario –Provided test descriptions but not test cases –Test lab is responsible for complete coverage

9. Software Diagnostics and Conformance Testing Votetest release strategy First release –Based on draft VVSG –“Basic test suite” –Tools and materials –Needs review and feedback Second release? –If consensus is that basic test suite is not enough –If there are problems to correct –Sync with finalized VVSG (if applicable) Maintenance and support –Keep up with VVSG maintenance (interpretations, errata) –Correct operational issues and coverage gaps as they arise

10. Software Diagnostics and Conformance Testing Votetest contents Data model that supports all draft VVSG voting variations SQL* schema that realizes the data model and the tabulation logic specified in the draft VVSG Test cases formalized as SQL scripts –We don’t know the interface to the test target –SQL used as surrogate language –Execute as written on the supplied database –Must be translated into whatever is required by the test target Report generator to display results from test oracle Expected test results Documentation Bonus: test generator * Schema uses extensions to ISO SQL

11. Software Diagnostics and Conformance Testing Votetest environment Test case execution Voting system environment Test case (SQL)Expected resultsReport generatorDatabase Test case (translated)Actual resultsVoting systemReport generator TranslateCompare

12. Software Diagnostics and Conformance Testing Usability of logic test tools and materials Technical expertise befitting an accredited test lab is assumed and required Test cases formalized as SQL scripts –More precise than informal test scripts –Automated translation is possible The expected output from each test case is provided as a plain text report –Test lab does not need to get the infrastructure to run on their machines to use the test scripts –Sanity check for running installations No huge up-front investment –Hardware requirements: one surplus PC –Software requirements: all free software

13. Software Diagnostics and Conformance Testing ###################################################################### BEGIN TEST CASE OUTPUT 2007-12-27 15:52:52-05 ###################################################################### $Id: 1-basic-1ofM.sql 415 2007-12-27 16:34:15Z dflater $ Small 1-of-M contest, no write-ins, no rejected ballots. Ballot styles: 1 Reporting contexts: 1 [... Integrity checks deleted...] [... View materialization log deleted...] ------------------------------------------------------------------------------- Report for context Precinct 1 generated 2007-12-27 15:52:52-0500 BALLOT COUNTS Configuration Read Counted ------------- ---- ------- Total 12 12 Blank 1 1 Precinct 1 Style 12 12 Blank 1 1 VOTE TOTALS President, vote for at most 1 Car Tay Fower 4 Tayra Tree 3 Beeso Tu 2 Oona Won 1 Nada Zayro 0 Overvotes 1 Undervotes 1 Counted ballots 12 Balance 0 ------------------------------------------------------------------------------- Report total volume: 76 - Includes optional reporting of blank ballots. - Excludes separate reporting of ballots cast vs. read. ###################################################################### END TEST CASE OUTPUT 2007-12-27 15:52:52-05 ###################################################################### Print header Reset database to baseline state Load test data Run integrity checks Generate report Print footer

14. Software Diagnostics and Conformance Testing The oracle Design requirement is correctness not performance Logic model of draft VVSG translated as transparently as possible into SQL views –Limited expressiveness of SQL means fewer ways to introduce faults (vs. programming) –Good news: the logic model itself translates with minimal overhead –Bad news: straight party voting and write-in reconciliation add a level of complexity Informal verification of correctness included in documentation Demonstrated scalability up to 2 million ballots Results of simple tests are manually confirmed Test suite + saved output + shell script = automated regression test X

15. Software Diagnostics and Conformance Testing Status as of 2008-01-23 3 baseline tests (no optional voting variations required) 19 single-variation tests covering 12 optional voting variations 66 two-variation tests covering 63 combinations of two voting variations –The other 3 combinations are not meaningful 1 three-variation test 3 tests based on sample ballots Total of 92 tests Working on documentation and presentation Could improve test generator and do more samples tests Needs NIST internal review, integration with other test efforts No public release yet

16. Software Diagnostics and Conformance Testing Challenges Can’t review prior art—everything claimed as trade secret Draft VVSG is a moving target—Standards and Advisory Boards Accretive release strategy—pressure to get it right the first time Realism—no two jurisdictions are alike Politics

17. Software Diagnostics and Conformance Testing Demo—Disclaimers For demonstration purposes only, we are about to execute a test case in an emulated environment This configuration has problems and is not recommended for production use The nonfatal error shown below should be ignored could not remove file or directory "base/55958": Directory not empty

18. Software Diagnostics and Conformance Testing Votetest environment Test case execution Voting system environment Test case (SQL)Expected resultsReport generatorDatabase Test case (translated)Actual resultsVoting systemReport generator TranslateCompare

19. THE DEM

20. Software Diagnostics and Conformance Testing End of presentation