1. TGDC Meeting, Jan 2011 VVSG 2.0 and Beyond: Usability and Accessibility Issues, Gaps, and Performance Tests Sharon Laskowski, PhD National Institute of Standards and Technology http://vote.nist.gov

2. TGDC Meeting, Jan 2011 Goal: Prioritize NIST usability and accessibility research for VVSG 2.0 VVSG 2.0 delivered to EAC in August 2007 This “baseline” VVSG 2.0 was an improvement to prior versions, but Voting systems, other standards, and our understanding of the needs of voters, poll workers and elections officials have evolved This briefing describes the usability and accessibility issues and gaps we have identified in the VVSG 2.0 requirements Page 2

3. TGDC Meeting, Jan 2011 Page 3 Paper ballots and motor disabilities Voting system must not require manually handling of the paper ballot e.g., automatic paper handling mechanism Level of Effort (LOE): Clarification of existing requirements

4. TGDC Meeting, Jan 2011 Paper ballot usability & accessibility Improve usability and accessibility of paper records for voters and poll workers LOE: Research and develop new requirements Page 4

5. TGDC Meeting, Jan 2011 Accessibility of verification * An accessible voting system must enable a voter to verify the content of a paper ballot using the same input and output features used to generate that ballot LOE: Clarification of existing requirements * Resolution will follow Page 5

6. TGDC Meeting, Jan 2011 Alternative input An accessible voting system must provide alternative input for voters with disabilities and this must provide equivalent access e.g., switch input LOE: Clarification of existing requirements Page 6

7. TGDC Meeting, Jan 2011 Harmonize with new accessibility technical standards Working with US Access Board, Update wheelchair reachability requirements to include motorized wheelchairs Harmonize with 2010 ADA and Section 508 refresh technical standards as they apply to voting systems LOE: Update requirements Page 7

8. TGDC Meeting, Jan 2011 Access for a wide range of functional limitations Goal is to maximize universal design requirements and increase personal assistive technology (PAT) capability PAT is not a substitute but an enhancement There are 4 specific areas to address LOE: Research and develop new requirements Page 8

9. TGDC Meeting, Jan 2011 Access for a wide range of functional limitations (continued) Develop requirements for PAT switch input jack and scanning interoperability Research range of manual dexterity capabilities and develop optimized set of requirements Research needs of aging voters and develop requirements Apply latest research in universal design to voting systems Page 9

10. TGDC Meeting, Jan 2011 Accessibility of PCOS Precinct count optical scanners (PCOS) use is growing PCOS now have more voter interface options, e.g., larger displays, paper feeding Are current VVSG 2.0 sufficient to ensure accessibility? LOE: Clarify and/or research and develop new requirements Page 10

11. TGDC Meeting, Jan 2011 Focus on access “features” VVSG organization by disability area rather than access features helped to simplify the organization, but it has led to some misinterpretation e.g., synchronized audio/visual is useful for multiple disabilities Usability section has universal design requirements Provide clarification and a way to more easily identify and pull out access features LOE: Clarification of requirements Page 11

12. TGDC Meeting, Jan 2011 Performance Tests: Overview The VVSG usability and accessibility requirements are, for the most part, design requirements Design requirements, unless they fully specific a user interface, are not sufficient to guarantee usability VVSG 2.0 has some requirements that are tested for pass/fail conformance with tests based on user performance Page 12

13. TGDC Meeting, Jan 2011 Performance tests A performance test is a usability test that is well- defined and repeatable Test performed with human participants Detects errors that one might observe when voters or poll workers interact with a system Detailed test protocol and test data (e.g., test ballot) Pass/fail criteria, such as benchmarks Distinguishes a system with good usability from a system with poor usability Feasible in terms of recruiting test participants and cost Page 13

14. TGDC Meeting, Jan 2011 VVSG 2.0 has 4 performance tests: VPP Voter Performance Protocol 3 benchmarks based on accuracy of the voters’ choices and completion 100 test participants Standard medium complexity ballot Narrow demographic, not using accessible features Status Shown face validity, repeatability, some reproducibility To do: obtain benchmark values for NIST calibration machine and verify procedural validity Page 14

15. TGDC Meeting, Jan 2011 Performance tests (continued) Research has shown that to get repeatability for testing with a wider demographic requires a much larger number of test participants High cost and impossible to recruit This suggested a hybrid approach Performance test Expert analysis of data to confirm major problems Page 15

16. TGDC Meeting, Jan 2011 Accessibility throughout the voting session Can voters with disabilities vote independently from the time they enter the voting booth through to casting their ballot? Status Verifying repeatability and procedural validity Gap: Adjusting demographics for people with dexterity disabilities Page 16

17. TGDC Meeting, Jan 2011 Documentation and poll worker usability Test is 4 teams each with 2 typical poll workers After a short training session, can they open, run, and close the polls using the documentation? Status Verifying repeatability and procedural validity Page 17

18. TGDC Meeting, Jan 2011 Performance test for accessible voting systems Using the hybrid approach, test with sufficient, but small numbers of people with disabilities to demonstrate usability Status: test protocol is currently under development and pilot testing Gap: what is a sufficiently wide range of disabilities to use, especially for dexterity? Page 18

19. TGDC Meeting, Jan 2011 Manufacturer testing Manufacturers are required to submit summative usability test reports using the standard ISO Common Industry Format for Usability Test Reports as part of their Technical Data Package for the test lab NIST CIF template and guide for voting systems Gap: tests must include a sufficiently wide range of functional limitations Page 19

20. TGDC Meeting, Jan 2011 Integrated accessible system VVSG 2.0 structured to require that the accessible system is one, single integrated system, but has been misinterpreted 1 accessible system per polling place, not separate components Addressing needs of people with 1 or more disabilities LOE: Clarification of requirements Page 20

21. TGDC Meeting, Jan 2011 Remote voting * Clarification of legal basis for accessibility requirements What is the scope of HAVA, UOCAVA, ADA and for States for accessibility of remote voting systems as it pertains to TGDC and NIST? What specific technical requirements should be developed for the range of voting outside of a polling place, absentee, and by mail? * resolution to follow Page 21

22. TGDC Meeting, Jan 2011 Proposed Resolution (1) White paper on “VVSG 2.0 and Beyond: Issues and Gaps in the Usability and Accessibility Requirements” The TGDC accepts the white paper titled “VVSG 2.0 and Beyond: Issues and Gaps in the Usability and Accessibility Requirements” for transmittal to the Election Assistance Commission. This paper responds to the directive of the Election Assistance Commission to the Technical Guidelines Development Committee (TGDC) Usability and Accessibility Working Group to continue ongoing research and requirements updates the VVSG 2.0 to the usability performance benchmarks already contained in the draft version of VVSG 2.0. Because the usability and accessibility requirements work together and cannot necessarily be isolated and analyzed as individual requirements, the Working Group identified research gaps that apply not just to the performance benchmarks but across the set of usability and accessibility requirements in the draft version of VVSG 2.0. Page 22

23. TGDC Meeting, Jan 2011 Proposed Resolution (2) Accessibility of Voter-Verifiable Paper Ballots To conform to HAVA Section 301 1, the TGDC identifies the following accessibility principles that will be used to guide recommendations for revisions to the VVSG. 1. An accessible voting system must provide mechanisms for private and independent voting in all three phases the voting process – generating a ballot, verifying the vote selections of that ballot, and casting the ballot -- regardless of the format of the ballot. 2. Accessibility requirements must not be lessened because the ballot format is paper and individuals with disabilities should not experience a decrease in their ability to vote privately and independently because of the use of a paper ballot. 1 HAVA Section 301 requires that a voting system be accessible for individuals with disabilities and provide the same opportunity for access and participation (including privacy and independence) as for other voters. It also requires that voters be able to verify (in a private an independent manner) the votes selected by the voter on the ballot before the ballot is cast and counted. Page 23

24. TGDC Meeting, Jan 2011 Proposed Resolution (2) Continued Accessibility of Voter-Verifiable Paper Ballots 3. Any vote casting process that leads to a determinative vote record (an official ballot of record or controlling record used in a recount) must be accessible (affirms the decision of the EAC in adopting the VVSG in 2005). a. In particular, if a determinative record paper ballot is voter-verifiable, the accessible system must enable a voter with a disability to verify the vote selections on the paper ballot through the same access features as were used to mark the vote selections (affirms Board of Advisors Resolution). b. A voter with a disability must not be required to manually handle a determinative record paper ballot in order to mark, verify or cast the ballot (affirms the decision of the EAC in adopting the VVSG in 2005). Page 24

25. TGDC Meeting, Jan 2011 Proposed Resolution (3) Accessibility requirements for voting systems used outside of a polling place HAVA Section 301 clearly requires that a voting system be accessible for individuals with disabilities and provide the same opportunity for access and participation (including privacy and independence) as for other voters. It also indicates that this requirement can be met by providing one accessible voting system per polling place. The VVSG requirements were developed for the one accessible voting system per polling place referenced in HAVA. The language of HAVA is less clear with regard to the accessibility requirements of voting that occurs outside of the polling place such as absentee voting and mail-in voting (done as a standard procedure or as part of UOCAVA voting). We assume that, in general, all voting systems must deliver some basic level of accessibility either under HAVA provisions or under the provisions of the ADA.(See 28 CFR 35.149 and 35.150). Specifically, the ADA requires equal program access which includes ensuring that communications with individuals with disabilities are as effective as communications with others through the delivery of appropriate auxiliary aids and services in a way that protects the privacy and independence of the individual with a disability. (See 28 CFR 35.160). The ADA definition of auxiliary aids is very broad and would seem to include accessible voting systems as developed to meet the VVSG requirements. (See 28 CFR 35.104, Definition of Auxiliary Aids and Services, paragraph 3.) Page 25

26. TGDC Meeting, Jan 2011 Proposed Resolution (3) continued Accessibility requirements for voting systems used outside of a polling place The TGDC recommends to the Elections Assistance Commission (EAC) that it provide clarification on the legal requirements for accessibility and specific direction to the TGDC to develop technical requirements for accessibility of voting systems used outside of a polling place including but not limited to mail-in and UOCAVA voting systems. Page 26