1. FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation  
Fenglong Ma1, Yaliang Li1, Qi Li1, Minghui Qiu2,
Jing Gao1, Shi Zhi3, Lu Su1, Bo Zhao4, Heng Ji5, Jiawei Han3
Presenter: Jing Gao
1SUNY Buffalo; 2Singapore Management University;
3University of Illinois Urbana-Champaign; 4LinkedIn;
5Rensselaer Polytechnic Institute

2. Which of these square numbers also happens to be the sum of two smaller numbers?16 25 36 49

3. A Straightforward Aggregation Method
Voting/Averaging
Take the value that is claimed by majority of the sources (users)
Or compute the mean of all the claims

4. Which of these square numbers also happens to be the sum of two smaller numbers?16 25 36 49

5. A Straightforward Aggregation Method
Voting/Averaging
Take the value that is claimed by majority of the sources (users)
Or compute the mean of all the claims
Limitation
Ignore source reliability (user expertise)
Source reliability
Is crucial for finding the true fact but unknown

6. Object
Aggregation
Source 1
Source 2
Source 3
Source 4
Source 5

7. Truth Discovery Principle
To learn users’ reliability degree and discover trustworthy information (i.e., the truths) from conflicting data provided by various users on the same object.
A user is reliable if it provides many pieces of true information
A piece of information is likely to be true if it is provided by many reliable users

8. Existing Work on Truth Discovery
Existing methods
Assign single expertise (reliability degree) to each user (source).
Expertise
Barack Obama
Albert Einstein
Michael Jackson

9. Example--Existing Truth Discovery Methods
Input
Question Set
User Set
Answer Set
Output
Users’ Expertise
Truths
Question
User u1 u2 u3 q1 1 2 q2 q3 q4 q5 q6
User u1 u2 u3
Expertise
5.00E-11
0.961
3.989
Question q1 q2 q3 q4 q5 q6
Truth 1 2
Question q1 q2 q3 q4 q5 q6
Ground Truth 1 2

10. Overview of Our Work Goal
To learn fine-grained (topical-level) user expertise and the truths from conflicting crowd-contributed answers.
Politics
Physics
Music

11. Example--Our Model Input Output Question Set User Set Answer Set
Word u1 u2 u3 q1 1 2 a b q2 c q3 q4 d e q5 f q6
Input
Question Set
User Set
Answer Set
Question Content
Output
Questions’ Topic
Topical-Level Users’ Expertise
Truths
Topic
Question K1 q1 q2 q3 K2 q4 q5 q6
User u1 u2 u3
Expertise K1
2.34
2.70E-4
1.00 K2
1.30E-4
2.35
Question q1 q2 q3 q4 q5 q6
Truth 1 2
Question q1 q2 q3 q4 q5 q6
Ground Truth 1 2

12. FaitCrowd Model Overview
Jointly modeling question content and users’ answers by introducing latent topics.
Modeling question content can help estimate reasonable user reliability, and in turn, modeling answers leads to the discovery of meaningful topics.
Learning topic-level user expertise, truths and topics simultaneously.

13. Modeling Question Content
Word Generation
Assume that each question is about a single topic (the length of each question is short).
Draw a topic indicator

14. Modeling Question Content
Word Generation
Assume that each question is about a single topic (the length of each question is short).
Draw a topic indicator
Assume that a word can be drawn from topical word distribution or background word distribution.
Draw a word category

15. Modeling Question Content
Word Generation
Assume that each question is about a single topic (the length of each question is short).
Draw a topic indicator
Assume that a word can be drawn from topical word distribution or background word distribution.
Draw a word category
Draw a word

16. Modeling Answers Answer Generation
The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
Draw user’s expertise

17. Modeling Answers Answer Generation
The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
Draw user’s expertise
Draw the truth

18. Modeling Answers Answer Generation
The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
Draw user’s expertise
Draw the truth
Draw the bias

19. Modeling Answers Answer Generation
The correctness of a user’s answer may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
Draw user’s expertise
Draw the truth
Draw the bias
Draw a user’s answer

20. Inference Method Gibbs-EM
Gibbs sampling to learn the hidden variables and
Gradient descent to learn hidden factors and

21. Datasets & Measure Datasets Measure The Game Dataset The SFV Dataset
Collected from a crowdsourcing platform via an Android App based on a TV game show “Who Wants to Be a Millionaire”.
2,103 questions, 37,029 sources, 214,849 answers and 12,995 words
The SFV Dataset
Extracted from Slot Filling Validation (SFV) task of the NITS Text Analysis Conference Knowledge Base Population (TAC-KBP) track.
328 questions, 18 sources, 2,538 answers and 5,587 words
Measure
Error Rate
The lower the better

22. Baseline Methods Basic Method Truth Discovery CrowdsourcingMV
Truth Discovery
Truth Finder
AccuPr
Investment
3-Estimates
CRH
CATD
Crowdsourcing
D&S
ZenCrowd
Variations of FaitCrowd
FaitCrowd-b
FaitCrowd-b-g

23. Performance Validation
Table 1: Performance on the Game Dataset.
Analysis
For easy questions (from Level 1 to Level 7), all the methods can estimate most answers correctly.
For difficult questions (from Level 8 to Level 10) , the performance of FaitCrowd is much better than that of the baseline methods.
FaitCrowd performs well on both Game and SFV datasets.
Table 2: Performance on the SFV Dataset.

24. Model Validation Goal Explanation
Illustrate the importance of joint modeling question content and answers by comparing with the method that conducts topic modeling and true answer inference separately.
Explanation
Dividing the whole dataset into sub-topical datasets will reduce the number of responses per topic, which leads to insufficient data for baseline approaches.
Table 3: Results of Model Validation.

25. Topical Expertise Validation
Goal
Validate the correctness of topical expertise learned by FaitCrowd.
Ideally, the expertise estimated by the proposed method is consistent with the ground truth accuracy.
Figure 1: Topic 2 on the Game Dataset.
Figure 2: Topic 4 on the SFV Dataset.

26. Expertise Diversity Analysis
Goal
Demonstrate that the topical expertise for each source varies on different topics.
Ideally, the topical expertise should correspond to the ground truth accuracy, i.e., the higher expertise, the higher the ground truth accuracy.
Figure 3: Source 7 on the Game Dataset.
Figure 4: Source 16 on the SFV Dataset.

27. Problem Solution Results Summary
Recognize the difference in source reliability among topics on the truth discovery task and propose to incorporate the estimation of fine grained reliability into truth discovery.
Solution
Propose a probabilistic model that simultaneously learns the topic-specific expertise for each source, aggregates true answers, and assigns topic labels to questions.
Results
Empirically show that the proposed model outperforms existing methods in multi-source aggregation with two real world datasets.

28. Thank you!
Questions?