1. OPENCOUNT IMPROVED SUPPORT FOR MACHINE- ASSISTED BALLOT-LEVEL AUDITS EVT/WOTE 2013. Washington DC. 8/13/2013. Eric Kim, Nicholas Carlini, Andrew Chang, George Yiu, Kai Wang, David Wagner University of California, Berkeley University of California, San Diego

2. Talk Overview Motivation How can OpenCount help the audit process? Challenges Important: Accuracy and scalability Pipeline Overview Election Experiences Questions

3. What is OpenCount? Software that tabulates elections Generates ballot-level cast vote records CVR 00001 President of the United States Mitt Romney Member, County Central Com. Shawn Nelson David John Shawver Greg Sebourn Steve Hwangbo

4. Motivation Want to perform a post-election audit Statistical ballot-level audit Risk-limiting audit Typically only have to examine tens to hundreds of ballots (depends on margin) More efficient than alternative CA: Each county hand-counts all ballots from1% of precincts

5. Motivation (cont.) Ballot-level audits require: access to the voting systems interpretation of each ballot Cast Vote Record (CVR) for each ballot Electronic record of the cast votes

6. Motivation (cont.) Cast Vote Record (CVR) CVR 00001 President of the United States Mitt Romney Member, County Central Com. Shawn Nelson David John Shawver Greg Sebourn Steve Hwangbo = ?

7. Motivation (cont.) Problem: current deployed voting systems do not output CVRs for each ballot Only output election totals

8. Motivation (cont.) Cant upgrade existing systems Most vendors are focusing on next-gen systems EAC certification process (U.S. Election Assistance Commission) would make upgrade expensive

9. Motivation (cont.) What is one to do? If you cant improve it, rebuild it!

10. OpenCount Tabulates elections Input: Scanned ballot images Output: Cast Vote Records, election totals. Built specifically with ballot-level audits in mind Open-source software (free!) http://code.google.com/p/opencount/

11. First Attempt: Blank Ballots Collect one blank ballot from each ballot style Blank Ballot: Unmarked ballot Style A Style B

12. With Blank Ballots… (1/6) Style A Style B

13. With Blank Ballots… (2/6) Style A Style B

14. With Blank Ballots… (3/6) Style A Style B

15. With Blank Ballots… (4/6) Style A Style B

16. With Blank Ballots… (5/6) Style A Style B

17. With Blank Ballots… (6/6) Style A Style B

18. Previous Work EVT/WOTE 2012 (Bellevue, Washington) First introduction of the OpenCount (2012) system Operator-Assisted Tabulation of Optical Scan Ballots. Kai Wang, Eric Kim, Nicholas Carlini, Ivan Motyashov, Daniel Nguyen, David Wagner. Required collecting all blank ballots

19. Previous Work (cont.) Problem: Did not scale to large elections Collecting blank ballots is a huge burden for election officials Blocked some counties from participating Overall, too much required effort

20. A Second Attempt New approach: No blank ballots

21. How can we do this?

22. No Blank Ballots Style A Style B

23. No Blank Ballots How to find: Voting Targets? Contests?

24. OpenCount Pipeline Overview of system Election experiences California risk-limiting audit pilot program

25. Scan Ballots (1/6) Use any commercial, off-the-shelf scanner

26. Ballot Grouping (2/6)

29. ~124,000 Ballots ~200 Styles

30. Ballot Grouping (2/6)

32. Implemented vendor-specific barcode decoders Diebold ES&S Hart Sequoia

33. Layout Annotation (3/6) Goal Specify location of contests and voting targets Perform data entry of contest text Only need to annotate one ballot from each style

34. Layout Annotation (3/6) How to find voting targets automatically?

35. Layout Annotation (3/6) 1.) User selects empty voting target

36. Layout Annotation (3/6) 1.) User selects empty voting target

37. Layout Annotation (3/6) Search for empty voting target on ballots Template Matching Grid-search Search for this:

38. Layout Annotation (3/6) Verify Matches

39. Layout Annotation (3/6) Problem: Voter marks interfere with template matching

40. Layout Annotation (3/6) Problem: Voter marks interfere with template matching Idea: Voters vote differently. Can find missing targets on other ballots with the same style

41. Layout Annotation (3/6) Ballot A Idea: Voters vote differently. Can find missing targets on other ballots with the same style

42. Layout Annotation (3/6) Ballot B Idea: Voters vote differently. Can find missing targets on other ballots with the same style

43. Layout Annotation (3/6) Union of detections from A + B Idea: Voters vote differently. Can find missing targets on other ballots with the same style

44. Layout Annotation (3.5/6) Contest text data entry Contest title, candidate names Judge of the Superior Court (Office No. 1) -Deborah J. Chuang -Eugene Jizhak

45. Layout Annotation (3.5/6) Cant rely completely on OCR Manually labeling each contest takes forever Number of distinct contests is small A few hundred at most Contests are duplicated on many ballot styles President of the US

46. Layout Annotation (3.5/6) Should only have to label this contest once!

47. Layout Annotation (3.5/6) Want to detect contest duplicates Simple idea: compare contest images Pixel-difference (L2 norm)

48. Layout Annotation (3.5/6) - Diff = 0.058 MATCH

49. Layout Annotation (3.5/6) - Diff = 0.175 NOT MATCH

50. Layout Annotation (3.5/6) Problem: contest visual appearance varies Word spacing, line wrapping, candidate re-ordering Different Line Wrap

51. Layout Annotation (3.5/6) - Diff = 0.146 NOT MATCH

52. Layout Annotation (3.5/6) Our approach: utilize OCR + edit-distance

53. Layout Annotation (3.5/6) Our approach: utilize OCR + edit-distance

54. Layout Annotation (3.5/6) Our approach: utilize OCR + edit-distance

55. Layout Annotation (3.5/6) Our approach: utilize OCR + edit-distance Match!

56. Ballot Interpretation (4/6) Goal Determine if voting targets are filled or empty FilledEmpty

57. Ballot Interpretation (4/6) Separating Line Sorted by Average Pixel Intensity

58. Ballot Interpretation (4/6)

61. Filled Empty

62. Generate CVRs (5/6) Output CVRs

63. Perform Audit (6/6) Finally, perform the audit! Done!

64. Election Experiences OpenCount has been used to support risk limiting pilot audits in several California counties Alameda, Madera, Merced, Napa, San Luis Obispo, Stanislaus, Ventura OpenCounts results matched all examined paper ballots perfectly

65. Election Experiences (cont.) County# Ballots# Ballot StylesTotal Time (2013) Stanislaus3,15117m 18s Merced7,120112m 31s Ventura17,301123m 6s Alameda1,374822m 1s San Luis Obispo 10,6892730m 35s Madera3,75716m 38s Napa6,809111h 56m 9s Yolo35,5326233h 36m

66. Election Experiences (cont.) County# Ballots# Ballot StylesTotal Time (2013) Speedup (2012 / 2013) Stanislaus3,15117m 18s2.40x Merced7,120112m 31s2.04x Ventura17,301123m 6s2.52x Alameda1,374822m 1s1.29x San Luis Obispo 10,6892730m 35s2.78x Madera3,75716m 38s1.28x Napa6,809111h 56m 9s2.78x Yolo35,5326233h 36m16.24x

67. Election Experiences (cont.) County# Ballots# Ballot StylesTotal Time (2013) Speedup (2012 / 2013) Stanislaus3,15117m 18s2.40x Merced7,120112m 31s2.04x Ventura17,301123m 6s2.52x Alameda1,374822m 1s1.29x San Luis Obispo 10,6892730m 35s2.78x Madera3,75716m 38s1.28x Napa6,809111h 56m 9s2.78x Yolo35,5326233h 36m16.24x

68. Election Experiences (cont.) County# Ballots# Ballot StylesTotal Time (2013) Human Time Marin29,12139811h 53m5h 45m Santa Cruz34,00413618h 50m5h 27m Leon124,20021614h 2s1h 53m Orange294,4021,8393d 22h 39s1d 8h 25m Previous version (2012) could not process elections of this size and complexity. Progress!

69. Conclusion Improvements to the OpenCount system Dont have to collect blank ballots Reduce operator effort significantly OpenCount is ready for election officials to use Used in ballot-level risk-limiting audits Audits made possible by OpenCount