1. TGDC Meeting, December 2011 Usability and Accessibility (U&A) Research Update Sharon J. Laskowski, Ph.D. http://vote.nist.gov

2. TGDC Meeting, December 2011 Outline U&A research goals Overview of all current research efforts Test method validation Tester qualifications, test documentation and workflow New test methods Improving standards Specific details Validation of Voter Performance Protocol Tester qualifications, test documentation and workflow Accessibility research Page 2

3. TGDC Meeting, December 2011 U&A Research Goals Improving test methods Test validation Tester qualifications, test documentation and workflow New test methods Improving standards Accessibility is the current focus See 2010 U&A Working Group paper on VVSG 2.0 U&A issues and gaps Page 3

4. TGDC Meeting, December 2011 Overview of Current Research Efforts Test method validation Voter Performance Protocol The “usability performance benchmarks” test from VVSG 2.0 U&A design inspection tests Accessibility throughout the voting session The “end-to-end accessibility” test Usability for poll workers test Page 4

5. TGDC Meeting, December 2011 Overview (Cont’d) Tester qualifications, test documentation and workflow New test methods Accessibility performance test Improving standards Dexterity research Audio-tactile interface research Page 5

6. TGDC Meeting, December 2011 Validation of Voter Performance Protocol (VPP) Goal: Assess the reproducibility of the VPP performance benchmark test in different geographic locations Data collection DRE and optical scanner 5 locations 125 voters/location Straight party vs. no straight party voting states Page 6

7. TGDC Meeting, December 2011 VPP (Cont’d) Locations DC Metro area – assessing reproducibility of earlier DC study Texas & Indiana – states with straight party voting Colorado & Tennessee – states w/o straight party Data analysis now under way Tennessee – more difficulties with completion on DREs Indiana – issues with recruiting to the specified test participant demographics Page 7

8. TGDC Meeting, December 2011 U&A Tester Qualifications Voting System Test Labs (VSTLs) need to select contractors who are qualified to execute U&A tests of the VVSG requirements National Voluntary Laboratory Accreditation Program (NVLAP) needs to assess U&A proficiency and skills for the accreditation of VSTLs We have developed assessments Based on our discussions with testers involved in our validation efforts and with the VSTLs Page 8

9. TGDC Meeting, December 2011 Assessment and Training for Testers Assessment factors – we have developed a checklist U&A expertise and experience, including testing participants with specific disabilities and assistive technology Skills in planning and performing tests Voting system domain knowledge, VVSG familiarity Most candidates will have strong skills on U&A testing, but weak domain knowledge VSTLs would be responsible for training their contractors on voting systems, testing to standards, etc. Page 9

10. TGDC Meeting, December 2011 Proficiency Test Questions NVLAP lab accreditation should include evaluating VSTL U&A testers’ proficiency in areas like: Voting system domain knowledge Familiarity with VVSG and test methods for U&A We have developed specific questions and a grading rubric, for example: How would you test 3.2.2.c “The voting system shall provide the voter the opportunity to correct the ballot for either an undervote or overvote before the ballot is cast and counted.” ? What do you do with the manufacturer's summative usability testing report in the TDP? What’s the difference between design inspections and performance tests? Page 10

11. TGDC Meeting, December 2011 Test Method Documentation Tester qualifications are not sufficient: test method documentation plays a significant role in tester performance Documentation usability: form and content Plain language Document navigation Multi-platform “Cookbook” quality: all supporting material together in a single package Scripts, screeners, protocols, NISTIRs, etc. Include ancillary sources along with the test methods Package for each major test method Page 11

12. TGDC Meeting, December 2011 Workflows for Test Method Execution Organize workflow by number of: Election rebuilds, testers, test participants Weigh multiple factors in designing workflow for efficiency Machine class and capabilities Ballot styles needed for a given test Coordination with other test teams, such as software or security, when conducting tests Need more information from VSTLs on how they conduct test campaigns Page 12

13. TGDC Meeting, December 2011 Development and Management of Test Methods In general: Need a feedback loop with VSTLs, EAC, NIST, NVLAP, election officials Lifecycle management of tests and all associated documentation Living standards process Need coordination across organizations Page 13

14. TGDC Meeting, December 2011 Accessibility Research NIST research supports incremental improvements to voting system standards Focus is on dexterity and audio-tactile interface research Examining research, best practices and standards in other domains and exploring how these apply to voting systems Performing small research studies to verify applicability EAC Accessible Voting Technology Initiative NIST providing technical support, especially for current state of the art, testing and evaluation Page 14

15. TGDC Meeting, December 2011 Discussion/Questions Page 15