1. Wisdom of Crowds and Rank Aggregation Wisdom of crowds phenomenon: aggregating over individuals in a group often leads to an estimate that is better than any of the individual estimates (e.g. Surowiecki, 2004) Goal: apply this idea to human ordering / ranking data: how can we aggregate the recollected orderings across individuals to best approximate some underlying ground truth? Approach: develop unsupervised Bayesian models for rank aggregation that take individual differences into account Experiment to collect human ordering data We tested 78 individuals on their ability to reconstruct from memory the order of items in 17 different tasks Example tasks: order of US presidents, the order of countries by landmass, the order of the ten commandments and the ten amendments. Performance was measured using Kendall’s Tau: The number of adjacent pair-wise swaps between recalled and true order. The Wisdom of Crowds in the Recollection of Order Information Mark Steyvers, Michael Lee, Brent Miller & Pernille Hemmer University of California, Irvine More information about our lab: http://psiexp.ss.uci.edu/research/madlab.htm http://psiexp.ss.uci.edu/research/madlab.htm do the experiments yourself at http://psiexp.ss.uci.edu/ http://psiexp.ss.uci.edu/ Example Raw Data Thurstonian Model v2: allowing Partial Knowledge Assumption: each individual has a unique variance (same for all items) but shares the same set of item means with the group. This model can represent varying degrees of “expertise” = 1 = 1+1 Ordering by Individual ABECD True Order ABCDE C D E ABAB ABCDEABCDE = 2 D A B C A B D C B A D CA C B D A D B C Generative Model ? ? latent ground truth Incorporate individual differences A. George Washington B. James Madison C. Andrew Jackson j individuals Strong wisdom of crowds effect across tasks Conclusion Using unsupervised Bayesian models for rank data, we can aggregate orderings across individuals such that aggregated ordering better approximates ground truth than any individual in the crowd: strong wisdom of crowd effect It is important to incorporate individual differences – some individuals are more expert than others. Models can estimate expertise levels in unsupervised fashion – individuals near consensus orderings are likely to be more expert (if individuals performed task independently) Mallows Model Distance-based model that assumes that observed orderings that are close to the group ordering are more likely than those far away. The probability of any observed order, given the group order is: Two-state model: an individual either produces an ordering according to a Mallows model (z=1) or a guessing process (z=0). We estimate the latent assignment z for each individual. This approach is related to Klementiev, Roth et al. 2009 Thurstonian Model (v1) Items are represented by coordinates on interval scale. Normal distributions represent uncertainty about item position – to order items, each individual draws one sample from each normal distribution and orders the items according to the samples. Means and standard deviations are shared among all individuals Individual differences: each individual is in one of two states: the Thurstonian state (z=1) and a guessing state (z=0) where there are no differences between items 78 individuals Thurstonian state (z=1) Guessing state (z=0) ordering by individual group ordering scaling parameter normalization constant Kendall tau distance function Experiment with 26 individuals ordering all 44 US presidents Mean Kendall tau averaged over all 17 tasks True ordering