1. Sahand Negahban Sewoong Oh Devavrat Shah Yale + UIUC + MIT

2. o Given partial preferences o Compute global ranking with scores to reflect intensity o Sports o Outcome of games between teams/players o Social recommendations o Ratings of few restaurants/movies o Competitive conference/Graduate admission o Ordering of few papers/applicants

3. o Partial preferences are revealed in different forms o Sports: Win and Loss o Social: Starred rating o Conferences: Scores o All can be viewed as pair-wise comparisons o IND beats AUS: IND > AUS o South Indies ***** vs MTR ***: SI > MTR o Ranking Paper 10/10 vs Other Paper 5/10: Ranking > Other

4. o Revealed preferences lead to o Bag of pair-wise comparisons o Sports, Social, Conferences, Transactions, etc. o Question of interest o Obtain global ranking over objects of interest o Teams/Players, Restaurants, Papers, Applicants. o Along with intensity/score for each object o Using given partial preferences/pair-wise comparisons

5. o Q1. Given weighted comparison graph G=(V, E, A) o Find ranking of/scores associated with objects o Q2. When possible (e.g. Conference/Crowd-Sourcing), choose G so as to o Minimize the number of comparisons required to find ranking/scores 1 62 3 4 5 A 12 A 21 # times 1 defeats 2

6. o We posit o Distribution over permutations as ground-truth o Pair-wise comparisons are drawn from this distribution DataDistributionRanking AB CB C A CB A CB A AB CB C A CB A 0.25 0.75 1 62 3 4 5 A 12 A 21

7. 1 62 3 4 5 A 12 A 21 o Input: complete preference (not comparisons) o Axiomatic impossibility [Arrow ’51] o Some algorithms o Kemeny optimal: minimize disagreements o Extended Condorcet Criteria o NP-hard, 2-approx algorithm [Dwork et al ’01] o Borda count: average position is score o Simple o Useful axiomatic properties [Young ‘74] 2 3 > 4 > 1 > 5 > 6 > 6 2 > 5 > 1 > 4 > 3 >

8. o Algorithm with comparisons o Variant of Kemeny optimal: o NP-hard o Variant of Borda count: average position from comparison? o If p ij = A ij /(A ij + A ji ) represent pair-wise marginal distribution o Then, Borda count is given as o Requires: G complete, many comparisons per pair o Also see (short list of relatd works): [Diaconis ‘87], [Alder et al ‘87], [Braverman-Mossel ’09], [Caramanis et al ‘11], [Fernoud et al ’11], [Duchi et al ‘12]… [Ammar, Shah ’11] 1 62 3 4 5 A 12 A 21

9. o General model o Effectively impossible to do aggregation o Practically o Restrict choice model o Popularly utilized model is instance of Thurstone’s ‘27 o Used for transportation system (cf. McFadden) o TrueSkill uses for ranking online gamers o Pricing in airline industry (cf. Talluri and Van Ryzin) o … 1 62 3 4 5 A 12 A 21

10. o Choice model (distribution over permutations) [Bradley-Terry-Luce (BTL) or MNL Model] o Each object i has an associated weight w i > 0 o When objects i and j are compared o P(i > j) = w i /(w i + w j ) o Sampling model o Edges E of graph G are selected o For each (i,j) ε E, sample k pair-wise comparisons

11. 1 62 3 4 5 A 12 A 21 o Random walk on comparison graph G=(V,E,A) o d = max (undirected) vertex degree of G o For each edge (i,j): o P ij = (A ji +1)/(A ij +A ji +2) x 1/(d+1) o For each node i: o P ii = 1- Σ j≠i P ij o Let G be connected o Let s be the unique stationary distribution of RW P o Ranking: o Use s as scores of objects o Closely related to Dwork et al ‘01 + Saaty ‘03

12. 1 62 3 4 5 A 12 A 21 o Random walk on comparison graph G=(V,E,A) o Let s be the unique stationary distribution of RW P o Ranking: o Use s as scores of objects o That is, object i has higher score if o It beats object j with higher score, o Or, beats many objects.

13. 1 62 3 4 5 A 12 A 21 o Random walk on comparison graph G=(V,E,A) o Let s be the unique stationary distribution of RW P o Ranking: o Use s as scores of objects o Compared to variant of Borda count:

15. o Error(s) = o G: Erdos-Renyi graph with edge prob. d/n d/n k

16. o Theorem 1 (Negahban-Oh-Shah). o Let R= (max ij w i /w j ). o Let G be Erdos-Renyi graph. o Under Rank centrality, with d = Ω(log n) o That is, sufficient to have O(R 5 n log n) samples o Optimal dependence on n for ER graph o Dependence on R ?

17. o Theorem 1 (Negahban-Oh-Shah). o Let R= (max ij w i /w j ). o Let G be Erdos-Renyi graph. o Under Rank centrality, with d = Ω(log n) o Information theoretic lower-bound: for any algorithm

18. o Theorem 2 (Negahban-Oh-Shah). o Let R= (max ij w i /w j ). o Let G be any connected graph: o L = D -1 E be it’s Laplacian o Δ = 1- λ max (L) o κ = d max /d min o Under Rank centrality, with kd = Ω(log n) o That is, number of samples required O(R 5 κ 2 n log n x Δ -2 ) o Graph structure plays role through it’s Laplacian

19. o Theorem 2 (Negahban-Oh-Shah). o Under Rank centrality, with kd = Ω(log n) o That is, number of samples required O(R 5 κ 2 n log n x Δ -2 ) o Choice of graph G o Subject to constraints, choose G so that o Spectral gap Δ is maximized o SDP [Boyd, Diaconis, Xiao ‘04]

22. o Bound on o Use of comparison theorem [Diaconis-Saloff Coste ‘94]++ o Bound on o Use of (modified) concentration of measure inequality for matrices o Finally, use this to further bound Error(s)

23. 1 62 3 4 5 A 12 A 21 o MIT admission system o ACM conferences (MobiHoc ‘11, Sigmetrics ‘13) o Past few years has been used for efficient reviewing o Daily polls (cf. A. Ammar) o polls.mit.edu o Netflix o ?

24. o Pair-wise comparisons o Universal way to look at partial preferences o Rank centrality o Simple and intuitive algorithm for rank aggregation o The comparison graph plays important role in aggregation o Choose G to maximize spectral gap of natural RW