Rank Aggregation Methods for the Web CS728 Lecture 11

Web Page Ranking Methods Reviewed PageRank – global link analysis Indegree – local link analysis HITS- topic-based link analysis Voting –NNN and Correlation Graph distance from seed URL length and depth Text-based methods (e.g., tf*idf)

Rank Aggregation BDCABDCA ABDCFEABDCFE BCDAFEBCDAFE “Consensus” ranking of all BDCAFEBDCAFE

Notations for Ranking Given a universe U, and ordered list τ of a subset of S of U τ=[x 1 ≥ x 2 ≥… ≥x d ], x i in S τ(i) : position of rank of i |τ|: number of elements full list : τ which contains all the elements in U partial list : rank only some of elements in U top d list : all d ranked elements are above all unranked elements Question: when are two orderings similar? Can you give a distance measure?

Measuring Distance Between Orderings Spearman’s Footrule Distance –σ, τ : two full list. –σ( i ) :rank of candidate i Kendall tau distance –Count the number of pairwise disagreements between the two lists

Example of Ordered-List Distance Example –S = {A,B,C,D,E} –σ, τ : two full list Spearman’s Footrule Distance –F(σ, τ ) = = 6 Kendall tau distance –K(σ, τ ) = |{(A,C), (B.D), (B,E), (D,E)}| = 4 ACEDBACEDB CABDECABDE τσ

Optimal ranking aggregation Optimality depends on the distance measure we use. Optimizing with Kendall tau distance, we obtain Kemeny optimal aggregation Can show satisfies neutrality and consistency –important properties of rank aggregation functions. Useful but computationally hard. Kemeny optimal aggregation is NP-hard. Will show that footrule-optimal is in P.

Two properties relate K and F For any full lists σ,τ K(σ,τ) ≤ F(σ,τ) ≤ 2 K(σ,τ) So we get a 2-approximation to Kemeny-optimality Since, if σ is the Kemeny optimal aggregation of full lists τ 1,…, τ k and σ’ optimizes the footrule aggregation then, K(σ’, τ 1,…, τ k ) ≤ 2 K(σ, τ 1,…, τ k )

Condorcet Criterion –An element of S which wins every other in pairwise simple majority voting should be ranked first. Extended Condorcet Criterion (XCC): –If most voters prefer candidate a to candidate b (i.e., # of i s.t.  i (a) <  i (b) is at least n/2), then also  should prefer a to b (i.e.,  (a) <  (b)). XCC is effective in ‘spam-fighting’ and thus good to use in meta-search. Condorcet Criteria and SPAM Filters

XCC: Not always realizable cba aab bcc  a) <  (b) <  (c) cba acb bac Not realizable

Voting Theory: Desired Properties Given set of candidates and voter preferences: seek an algorithm that ranks candidates which satisfies a set of desired properties Which combination of properties are realizable? 1) Independence from Irrelevant Alternatives: Relative order of a and b in  should depend only on relative order of a and b in  1,…,  n. –Ex: if  i = (a b c) changes to (a c b), relative order of a,b in  should not change.

Desired Properties: 2) Neutrality No candidate should be favored to others. –If two candidates switch positions in  1,…,  n, they should switch positions also in . 3) Anonymity No voter should be favored to others. –If two voters switch their orderings,  should remain the same.

Desired Properties: 4) Monotonicity If the ranking of a candidate is improved by a voter, its ranking in  can only improve. 5) Consistency If voters are split into two disjoint sets, S and T, and both the aggregation of voters in S and the aggregation of voters in T prefer a to b, then also the aggregation of all voters should prefer a to b.

Desired Properties 6) No Dictatorship: f(  1,…,  n ) !=  I 7) Unanimity (a.k.a. Pareto optimality): If all voters prefer candidate a to candidate b (i.e.,  i (a) <  i (b) for all i), then also  should prefer a to b (i.e.,  (a) <  (b)).

Desired Properties 8) Democracy: satisfies extended Condorcet Criterion XCC. –Always works for m = 2. –Not always realizable for m ≥ 3. Theorem [May, 1952]: For m = 2, Democracy is the only rank aggregation function which is monotone, neutral, and anonymous.

Arrow’s Impossibility Theorem [Arrow, 1951] Theorem: If m ≥ 3, then the only rank aggregation function that is unanimous and independent from irrelevant alternatives is dictatorship. –Won Nobel prize (1972)

Borda’s method Easy and intuitive - Several “score-based”variants; 1781 Violates independence from irrelevant alternatives C3 C1. C7 C8 C10 C7 C1. C8 C3 C10 C3 C2. C7 C10 C9 C3 C8. C1 C15 C10  1  2  3  4 B i (c)=the number of candidates ranked below c in  i B (c)=  i B i (c) Sorted in decreasing order B i (C8) =

Partial lists Handle partial lists by giving all the excess scores equally among all unranked candidates, Example: Candidates number = 100 Ranked candidates number =70 (score: 31~100) =>Assign score 31/30 to each 30 unranked candidates

Footrule optimal aggregation Footrule optimal aggregation can be computed in polynomial time. is a good approximation of Kemeny optimal aggregation. Proof : Via minimum cost perfect matching

Markov Chain method for rank aggregation. States=candidates Transitions depend on the preference orders given by voters Basic idea: probabilistically switch to a “better candidate” Rank candidates based on stationary probabilities!

Markov chain advantages Handling partial list and top d list by using available comparisons to infer new ones Handling uneven comparison and list length Computation efficiency –O(NK) preprocessing,O(K) per step for about O(N) steps

Four ways to build transition Matrix Current state is candidate a. MC1: Choose uniformly from multiset of all candidates that were ranked at least as high as a by some voter. – Probability to stay at a: ~ average rank of a. MC2: Choose a voter i uniformly at random and pick uniformly at random from among the candidates that the i-th voter ranked at least as high as a. MC3: Choose a voter i uniformly at random and pick uniformly at random a candidate b. If i-th voter ranked b higher than a, go to b. Otherwise, stay in a. MC4: Choose a candidate b uniformly at random If most voters ranked b higher than a, go to b. Otherwise, stay in a. – Rank of a ~ # of “pairwise contests” a wins.

A locally Kemeny optimal aggregation is a relaxation of Kemeny Optimality A locally Kemeny optimal aggregation satisfies the extended Condorcet property and can be computed in “kO(nlogn)” worst case, O(n 2 ) Many of existing aggregation methods do not satisfy ECC. =>Given τ 1, …,τ k use your favorite aggregation method to obtain a full list μ. And Apply local kemenization to μ with respect to τ 1, …,τ k.

A local Kemenization of a full list with respect to Compute a locally Kemeny optimal aggregation of that is maximally consistent with This approach: (1) preserves the strengths of the initial aggregation. (2) ranks non-spam above spam. (3) gives a result that disagrees with on any pair ( i, j ) only if a majority of the τ’s endorse this disagreement. (4) for every d, 1 ≤ d ≤ | μ |, the restriction of the output is a local Kemenization of the top d elements of μ Local Kemenization is a procedure to get locally Kemeny optimal aggregation.

How do we perform local kemenization? ABFECDABFECD BCAEFDBCAEFD ACFDEBACFDEB BFDCAEBFDCAE CABFEDCABFED BADCEFBADCEF BBABA ABAB ABDABD ABDCABDC ABCDABCD ABCFEDABCFED Local Kemenization Example! disagree A>B: 3 A<B: 2 B>D: 4 B<D: 1

Experiments: meta-search K = Kendall distance SF = scaled footrule distance IF = induced footrule distance LK = Local Kemenization