1. Improving the Usability and Accessibility of Voting Systems and Products Dr. Sharon Laskowski sharon.laskowski@nist.gov July 9, 2004 TDGC Meeting

2. 2

3. 3 Human Factors/Usability Perspective on Voting Systems: Voters Cognitive and physical nature of the voters Physical environment Psychological environment Voting product Usability is determined by the demands of the system and the voter’s ability to perform under those demands

4. 4 Measuring Accessibility and Usability Accessibility  The degree to which a system is available to and usable by individuals with disabilities Usability  A measure of the effectiveness, efficiency, and satisfaction achieved by a specified set of users performing specified tasks with a given product Metrics: errors causing a vote cast not as intended or a vote not cast, (errors prior to success), and time to cast vote Designing and measuring process  User-centered design  Diagnostic usability evaluation  Testing performance—usability testing

5. 5 State of Usability of US Voting Systems In general, voting systems have not been measured for usability nor have they been developed using a user-centered design process We do not know the degree to which voters cast their vote NOT as they intended due to confusion with the user interface Note observations by, CalTech/MIT Voting Technology Project, Herrnson et.al., and others such as Doug Jones

6. 6 Design and Performance Standards Design Standards—how the product is designed  For example, font size, ballot instructions Performance Standards—how the product functions  No overvoting, test by demonstration  Time to cast vote, failures in casting vote as intended Requires: measuring with users against benchmarks, Sample ballots of different complexity, and Well-defined test protocols and user groups

7. 7 We currently cannot measure usability of voting systems  E.g., select/deselect  Need high degree of usability  Following design guidelines does not necessarily insure usability  Usability engineering provides measurement methods, but not necessarily to the degree we need specifically for voting We need standards and conformance tests that do measure degree of usability and accessibility, if systems are going to be qualified and certified for usability and accessibility Measurement for Qualification and Certification rigorous research & experiments informal evaluation Easy, variableComplex, reliable feasible reproducible conformance testing

8. 8 Current Voting Standards and Testing Current VSS has some accessibility standards, but only a usability appendix ITAs currently perform qualification tests Can we test for usability and accessibility?  Standards must be clear, unambiguous, testable  Requires procedures for testing the voting product against the standards (conformance testing) For example, inspection, demonstration, operation IEEE P1583 draft standards  Task Group 3 has made some progress But, lack of resources and small vendor base have been a barrier to developing standards that are performance-based standards, benchmarks and conformance tests

9. 9 The HF Report Our report recommends an approach that will produce measurable voting systems standards for usability and accessibility  Doesn’t need a lot of research  Does need: Expertise in conformance test development Some applied research to develop user testing protocols Neutral third parties to accomplish this No cheap, quick fixes  Could require some usability testing to avoid major usability blunders, but this is no guarantee

10. 10 10 Recommendations 1. Performance-based, high-level usability standards 2. Complete set of user-related functional requirements 3. Avoid low-level design specifications. Use only product design requirements that have been validated as necessary 4. Applied research to support the development of usability and accessibility standards 5. Review current requirements (Access Board, the current VSS, draft IEEE standards) for possible adoption 6. Ballot design guidelines 7. Guidelines for facility and equipment layout; design and usability testing guidelines for vendor- and state-supplied documentation and training materials 8. Vendors should incorporate a user-centered design approach 9. Conformance tests for voting products against the applicable accessibility requirements. 10. Valid, reliable, repeatable, and reproducible process for usability conformance testing of voting products against the standards described in recommendation 1) with agreed upon usability pass/fail requirements.

11. 11 Most Critical Need A set of usability standards for voting systems that are performance-based, with  Objective measures  Conformance test procedures Then voting products and systems can be certified that they meet the usability standards This is the only way to guarantee high levels of usability

12. 12 RoadMap (Details in Report) Short term: encourage usability and user- centered design Long term:  Use best of IEEE and other standards …and ballot design guidance  Develop user test procedures  Collect user data to define performance baselines  Develop performance standards and conformance tests