Using Informative Priors to Enhance Wisdom in Small Crowds

Eyewitness testimony Common problems Reliance on error-prone eyewitnesses Reliance on very small number of eyewitnesses (often one) This research Integrating recalled memories across multiple individuals Focus on situations involving small number of individuals with potentially poor memory Approach Incorporate informative priors into a wisdom of crowd aggregation model Prior knowledge is extracted from a different group of individuals 2

Illustrative example: Galton’s Ox Galton’s original experiment estimate weight of ox (true answer is 1198 pounds) large number of individuals (800) median answer came within 9 pounds of true weight Thought experiment Suppose we only have small numbers of individuals Suppose we know something about ox weight a priori 3

Galton’s Ox with Prior Information 4 j=1,..,N Goal: infer the true mean from the observed estimates Assumptions:

Bayesian Inference Standard solution: Measure mean absolute difference 5

Simulation Vary the number of individuals (N) and strength of the assumed prior by the researcher (σ 0 ) 6 uninformative prior (based on 50,000 repetitions of Galton’s experiment in each cell)

Application to Episodic Memory We apply the same ideas to an episodic memory task: serial recall Serial recall experiment (N=28) Study event sequence in video Order the test images from memory Norming experiment (N=16) No study phase Order test images in as natural order as possible Allows us to build a model for prior probability of each sequence 7

Materials Type I 3 videos with stereotyped event sequences (e.g. wedding) associated with “strong prior” Type II 3 videos with less predictable event sequence associated with “weak prior” Extracted 10 images for testing 8

Example Type I Sequence: Bus video 9 priorserial recall Note: first row is subject id, second row is Kendall tau to true ordering

Example Type II Sequence: Clay Video 10 priorserial recall

Empirical Results: distribution of tau distances 11 Note: bin width is two so first bar is the relative proportion to tau = 0 and tau = 1 Type I Type II

Mallows Model 12 Kendall tau distance Scaling parameter Normalizing constant Latent truth Recalled order

ωyjyj θjθj j=1,..,M Model 1: Mallows Model with Uninformative Prior

ωyjyj θjθj j=1,..,M θ*θ* ωoωo yojyoj θojθoj j=1,..,N Model 2: Mallows Model with Informative Prior (prior knowledge data) (memory data)

Example calibration result 15 (clay video; all subs) Model 1 Model 2

Wisdom of Crowds Effect (Model 1) 16

Pick worst K individuals (from episodic task) 17 (means of taus across mcmc samples; averaged across 3 videos for each type)

Modeling Results (pick worst K individuals; means of taus across mcmc samples) 18

Pick random K individuals 19 (means of taus across mcmc samples; averaged across 3 videos for each type)

Pick random K individuals 20 means of taus across mcmc samples

Left over 21

Pick worst K individuals (single tau) 22

Modeling Results (pick worst K individuals; single tau – use Borda count to aggregate samples) 23

This research Wisdom of crowds Recover original sequence of events based on recollected order from a group of individuals Prior knowledge We measure prior knowledge about event sequences Incorporate this prior knowledge into our wisdom of crowd aggregation model  informative priors 24