1. Incentivize Crowd Labeling under Budget Constraint
Qi Zhang, Yutian Wen, Xiaohua Tian,
Xiaoying Gan, Xinbing Wang
Shanghai Jiao Tong University, China

2. Outline Introduction to Crowdsourcing Mechanism
Problem Formulation and Mechanism Setting
Mechanism Analysis
Performance Evaluation 2

3. Background Crowdsourcing systems leverage human wisdom to
perform tasks, such as:
Image classification
Character recognition
Data collection

4. Types of Tasks Tasks can be divided into two categories:
Structured response format
Binary choice
Multiple choice
Real Value
Unstructured response format
Logo design

5. Motivating Example Workers Task Dog Dog Dog Cat Cat Crowdsourcing
Example: Image classification
Workers
Task
Dog
Dog
Inference
Algorithm
Cat
Dog
Cat
Allocation
Crowdsourcing
Platform
Dog

6. (1)Tasks (2)Bids Framework: Reverse Auction (5)Answers (6)Payments
(3)Winning bids determination
(4)Winning bids
(5)Answers
(6)Payments

7. Major Challenges(1) To design a successful crowdsourcing system
Task Allocation (winning bids)
Tasks should be allocated evenly
Payment Determination:
Must provide proper incentives (monetary rewards)
Inference Algorithm:
Should improve overall accuracy
Should address the diversity of the crowd

8. Major Challenges(2) We need to model on Diverse task difficulty
Dog or Cat
Older than 30 or Not
Diverse worker quality
Cat

9. We focus on binary choice tasks
Model on Tasks(1)
We focus on binary choice tasks
Each task is a 0 – 1 question
(Assumption) Each worker is uniformly reliable
Task Soft Label
Probability that the task is labeled as 1( by a reliable worker)
Crowd Label
0 or 1

10. Update parameters (a,b) by Bayes rule
Model on Tasks(2)
The soft label is viewed as a random variable drawn from Beta distribution
Update parameters (a,b) by Bayes rule
Inference
The task is inferred as 1
Prior Parameters
Posterior
Prior
Likelihood
More than half agree

11. Framework: Reverse Auction
The platform publicizes a set of binary tasks
Workers reply with a set of bids
Each bid is a task-price pair
(Allocation) The platform sequentially decide winning bids
(Payment) Winning workers provide labels and get payment

12. Crowdsourcing Platform Utility
After observing all crowd labels , the distribution is updated as
Platform Utility: KL Divergence between
the initial and the final distribution

13. Problem Formulation We want:
Platform utility maximization under budget constraint
Individual rationality
Truthful about the cost
Truthful bid Untruthful bid
Computation Efficiency

14. Allocation Scheme (1) All bids Candidate Remaining bids Discard
The task allocation(winning bid determination) is sequential :
Candidate selection
one candidate a round
Proportional rule check
Answer collection & Soft label update
The allocation scheme repeats the 3 steps until
Candidate
Remaining bids
Discard
Winning bid
All bids
Discard
Winning bids

15. Allocation Scheme (2) The candidate selection is greedy PU Gain
The largest platform utility gain per unit price
Platform utility gain:
PU Gain
Candidate
Price
Current distribution
Updated distribution

16. Allocation Scheme (3) Proportional rule check Soft label update
Collect the answer from the winning bid
Update the soft label according to Bayes rule
budget
price
fraction ratio

17. Allocation Scheme (5) Computationally efficient ! Candidate selection
Proportional rule check
Soft label update
Computationally efficient !

18. Payment Scheme(1) {A, B, C} {D, E, F} {A, B, D, E} {F}
p(C) = max {b1,b2, b3, b4}
Winning bids
{A, B, C}
Discard
{D, E, F}
Kick out C
{ A,B,D,E,F }
Winning bids
{A, B, D, E}
Discard
{F} b2 b3 b1 b4
b1 is the minimum price
that bid C can replace bid A C

19. Payment Scheme(2)
(Proposition)The winning bid C is paid threshold payment.
p(C) C’s payment, b(C) C’s bid
if b(C) < p(C), C is a winning bid
if b(C) > p(C), C is discarded
p(C)=max { b1, b2, b3, b4}
Winning bids
{A, B, D, E} b2 b3 b1 b4 C

20. Payment Scheme(3) (Proposition)The incentive mechanism is truthful
Each bid has a cost
Workers will truthfully reveal the cost as asked price
Why?
Proof: Threshold payment + Greedy candidate Selection

21. Individual Rationality
(Proposition)The incentive mechanism is individual rational
The utility of a winning bid is nonnegative
Proof： Let us consider the winning bid C
C is the 3rd winning bid.
The first 2 bids are the same
b3 is the minimum price
that bid C can replace the new 3rd bid (D)
It is true that b3 > b(c) ！
p(C) = max {b1, b2, b3, b4}, p(C) > b3
p(C) > b(C)
New Winning bids
{A, B, D, E} b2 b3 b1 b4
{ A, B, C}
Original wining bids

22. Budget Feasibility
(Proposition, Payment Bound) Payment to each winning bid
is upper bounded by
Proportional rule:
Set

23. Performance Evaluation(1)
Benchmark
Untruthful Allocation: Workers’ cost is public information
Random Allocation: Candidate selection is random
Truthful
Running Time
Platform Utility
Benchmark 1 
High
Benchmark 2 
Low
My Mechanism

24. Performance Evaluation(2)
Metric 1 : Platform Utility
Platform utility vs. Budget
Price of Truthfulness
Gain over random allocation

25. Performance Evaluation(3)
Metric 2 : Budget Utilization
Payment / Budget
Budget utilization gain
Over random allocation

26. Thank you !
Presented by : Qi Zhang