1. Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs Tyler Lu and Craig Boutilier University of Toronto

2. Introduction New communication platforms can transform the way people make group decisions. How can computational social choice realize this shift? Choices People Computational Social Choice Consensus 2

3. Introduction Computational social choice – Aggregate full preferences (rankings) – Mostly study rank-based schemes (Borda, maximin, etc…) Rank-based voting schemes rarely used in practice Problem: Cognitive and communication burden Our approach (recent work): Elicit just the right preferences to make good enough group decisions This work: Multi-round elicitation and probabilistic preference models to further reduce burdens Alice BobCindy > 1 2 1 3

4. Outline Preliminaries Multi-round Probabilistic Vote Elicitation Methodology and Analysis for One-round Experimental Results 4

5. Preliminaries Voters N = {1..n}; alternatives/items A = {a 1 …a m } Vote v i is a ranking of A Complete profile v = (v 1, …, v n ) Alice Bob 1 2 3 voting rule r 5 Cindy

6. Score-based Rules Many rules have score-based interpretation – Surrogate for total group satisfaction – E.g. Borda, Bucklin, maximin, Copeland, etc… Associates a score for each item given full rankings s(a, v) Winner has highest score 6 s(, v) = 7 s(, v) = 6 s(, v) = 5 Alice Bob 1 2 3 Cindy Borda scores

7. Partial Preferences Partial vote p i is a partial order of A – Represented as a (consistent) set of pairwise comparisons – Higher order: top-k, bottom-k, … – Easy for humans to specify Partial profile p 7 Alice > >> How to make decision with partial preferences?

8. Decision with Partial Preferences Possible and necessary co-winners [Konczak, Lang05] Recently: minimax regret (MMR) [Lu, Boutilier11] – Provides worst-case guarantee on score loss w.r.t. true winner – Small MMR means good enough decision – Zero MMR means decision is optimal 8

9. Minimax Regret 9 Adversarial Best response

10. Vote Elicitation MMR: good choices with right partial votes – How to minimize amount of partial preference queries to make good decision? MMR-based incremental elicitation [Lu, Boutilier11] – Problem: must wait for response before next query 10

11. Incremental Elicitation Woes Each query is a (voter, pairwise comparison) pair – Exploits MMR, depends on all previous responses 11 Elicitor YES NO … > ? > ? … Bob annoyed at having to come back to answer query interruption cost

12. Our Solution: Multi-Round Batching Send queries to many voters in each round 12 Elicitor Round: 1 Give your top 2 1. Round: 2 Give your next top 1 3. MMR ε Recommendation: Interruption cost reduced 2.

13. Multi-Round Probabilistic Vote Elicitation Query class: rank top-5, is A > B?, etc… – Single request of preferences from voter – Have different cognitive costs In each round π selects a subset of voters, and corresponding queries – Can be conditioned on previous round responses Function ω, selects winner and stops elicitation How to design elicitation protocol with provably good performance? – Worst-case not useful (for common rules) – Use probabilistic preference models to guide design 13

14. Multi-Round Probabilistic Vote Elicitation Distribution P over vote profiles – Induced distribution over runs of protocol (π, ω) Can define distribution over performance metrics 14 Quality of winner: Max regret, expected regret Amount of information elicited: equivalent #pairwise comparisons, or bits. Number of rounds of elicitation Tradeoffs! Depends on what costs are important.

15. One-Round Protocol Query type: top-k – Rank your top-k most preferred Simple top-k heuristics [Kalech et al11] – Necessary and possible co-winners – No theoretical guarantees on winner quality – Dont provide guidance on good k – No tradeoff between winner quality and k 15

16. Probably Approximately Correct (PAC) One-Round Protocol Any rank-based voting rule Any distribution P over profiles What is a good k? – p[k] are partial votes after eliciting top-k k*: smallest k, with prob. 1 - δ, MMR(p[k]) ε As long as we can sample from P, we can find approximately good k… – Samples can come from historical datasets, surveying, or generated from learned distribution 16

17. Probably Approximately Correct One-Round Protocol General Methodology Input: sample of vote profiles: v 1, …, v t MMR accuracy ε > 0 MMR confidence δ > 0 Sampling accuracy ξ > 0 Sampling confidence η > 0 Find best the smallest k with 17

18. Probably Approximately Correct One-Round Protocol Theorem: if sample size then for any P, with probability 1 - η, we have (a) k* (b)P[ MMR(p[ ]) ε ] 1 - δ - 2ξ 18

19. Practical Considerations Sample size from theorem typically unnecessarily large Empirical methodology can be used heuristically Can generate histograms of MMR for profile samples from runs of elicitation – Can eyeball a good k – Can eyeball tradeoffs with MMR 19

20. Experimental Results First experiments with Mallows distribution – Rankings generated i.i.d. – Unimodal, with dispersion parameter – t = 100 profiles (for guarantees, use bounds for t) Borda voting Simulate runs of elicitation – Measure max regret and true regret – Normalize regret by number of voters 20

21. Experimental Results 21 x-axis is MMR per voter

22. Experimental Results 22

23. Experimental Results 23

24. Experimental Results 24

25. Experimental Results 25

26. Experimental Results 26 Sushi 10 alternatives 50 profiles, each with 100 rankings

27. Experimental Results 27 Dublin North 12 alternatives 73 profiles, each with 50 rankings

28. Concluding Summary Model of multi-round elicitation protocol – Highlights tradeoffs between quality of winner, amount of information elicited, and #rounds – Probabilistic preference profiles to guide design and performance instead of worst-case One-round, top-k elicitation – Simple, efficient empirical methodology for choosing k – PAC guarantees and sample complexity – With MMR solution concept, enables probabilistic and anytime guarantees previous works cannot achieve 28

29. Future Work Multi-round elicitation, top-k or pairwise comparisons Fully explore above tradeoffs (associative different costs) Assess expected regret and max regret 29

30. The End 30