Crowdsourcing with Multi- Dimensional Trust Xiangyang Liu 1, He He 2, and John S. Baras 1 1 Institute for Systems Research and Department of Electrical and Computer Engineering University of Maryland, College Park, MD 2 Deptment of Computer Science, University of Maryland, College Park, MD

Crowdsourcing Background Crowdsourcing Assignment Engine Malicious workers More reliable workers Pure experts Amazon Turkers Trust Evaluation True Label Inference clients Upload tasks Estimated answers

Motivation Tasks on crowdsourcing markets like Amazon Mechanical Turk often require knowledge in widely-ranging domains. Workers have different level of reliability in different domains. Goal: design algorithm to jointly evaluate workers’ trust values in each of the domains and at the same time estimate true labels for classification crowdsourcing tasks. Task politics sports fashion worker [good, bad, bad] worker [bad, good, bad] [bad, bad, good]

Notations Domain distribution for question i Domain for question i Truth label for question i. Take value from {0, 1} Trust vector for worker j Answer given by worker j to question i. Takes value from {0, 1} Hyper parameter of the Dirichlet prior on domain distribution. Parameter of the beta prior on trust of workers Probability that question i is associated with lth domain Trust value for worker j in domain l. Takes value from [0,1]

Probabilistic Graphical Model: No Feature Compute posterior probability for trust and true label.

Inference and Estimation Obtain the approximate posterior distributions by maximizing the lower bound of the log likelihood: We update the trust and true labels as below:

Probabilistic Graphical Model With Features Compute posterior probability for trust and true label.

Inference and Estimation Obtain the approximate posterior distributions by maximizing the lower bound of the log likelihood:  E-Step: given current model parameter estimation,, obtain approximate posterior q.  M-Step: given current posterior q, calculate the new model parameter estimation by maximizing lower bound

Probabilistic Graphical Model With Topic Models Multi-dimension trust crowdsourcing Topic model

Inference and Estimation Alternatively update approximate posterior distribution for different hidden variables:

Experiments Worker TypeDomain 0Domain 1 Type 10.5 Type Type Type PimaMVSDCMDFCMDC (1,2,2,1) N/A (2,2,2,1) N/A (3,2,2,1) N/A (1,2,2,1)NF N/A0.039 (2,2,2,1)NF N/A0.043 (3,2,2,1)NF N/A0.041

Experiments

Scientific TextMVMDCMDTC T T T T T T Tested model on 1000 scientific text annotated by five workers. Each worker answers whether a given sentence contains contradicting statements. Each sentence has the text data along with the labels provided by the five experts. We simulate D workers in total where worker j answers questions from topic j perfectly and answers questions from topics other than j close to randomly.

Experiments To show that MDTC can recover workers’ trust in each of the domains, we plot the mean trust value of 8 workers in each of the eight domains.

Conclusions Formulated a probabilistic graphical model with multi- dimensional characteristics and provided novel inference method based on variational inference. (MDC) The model is flexible and easily extensible to incorporate feature values. (MDFC) We extended MDC with topic discovery based on questions’ text descriptions and derive an analytical solution to the collection variational inference.

Thank you