Download presentation

Published byJahiem Bess Modified over 2 years ago

1
**Incentivize Crowd Labeling under Budget Constraint**

Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China

2
**Outline Introduction to Crowdsourcing Mechanism**

Problem Formulation and Mechanism Setting Mechanism Analysis Performance Evaluation 2

3
**Background Crowdsourcing systems leverage human wisdom to**

perform tasks, such as: Image classification Character recognition Data collection

4
**Types of Tasks Tasks can be divided into two categories:**

Structured response format Binary choice Multiple choice Real Value Unstructured response format Logo design

5
**Motivating Example Workers Task Dog Dog Dog Cat Cat Crowdsourcing**

Example: Image classification Workers Task Dog Dog Inference Algorithm Cat Dog Cat Allocation Crowdsourcing Platform Dog

6
**(1)Tasks (2)Bids Framework: Reverse Auction (5)Answers (6)Payments**

(3)Winning bids determination (4)Winning bids (5)Answers (6)Payments

7
**Major Challenges(1) To design a successful crowdsourcing system**

Task Allocation (winning bids) Tasks should be allocated evenly Payment Determination: Must provide proper incentives (monetary rewards) Inference Algorithm: Should improve overall accuracy Should address the diversity of the crowd

8
**Major Challenges(2) We need to model on Diverse task difficulty**

Dog or Cat Older than 30 or Not Diverse worker quality Cat

9
**We focus on binary choice tasks**

Model on Tasks(1) We focus on binary choice tasks Each task is a 0 – 1 question (Assumption) Each worker is uniformly reliable Task Soft Label Probability that the task is labeled as 1( by a reliable worker) Crowd Label 0 or 1

10
**Update parameters (a,b) by Bayes rule**

Model on Tasks(2) The soft label is viewed as a random variable drawn from Beta distribution Update parameters (a,b) by Bayes rule Inference The task is inferred as 1 Prior Parameters Posterior Prior Likelihood More than half agree

11
**Framework: Reverse Auction**

The platform publicizes a set of binary tasks Workers reply with a set of bids Each bid is a task-price pair (Allocation) The platform sequentially decide winning bids (Payment) Winning workers provide labels and get payment

12
**Crowdsourcing Platform Utility**

After observing all crowd labels , the distribution is updated as Platform Utility: KL Divergence between the initial and the final distribution

13
**Problem Formulation We want:**

Platform utility maximization under budget constraint Individual rationality Truthful about the cost Truthful bid Untruthful bid Computation Efficiency

14
**Allocation Scheme (1) All bids Candidate Remaining bids Discard**

The task allocation(winning bid determination) is sequential : Candidate selection one candidate a round Proportional rule check Answer collection & Soft label update The allocation scheme repeats the 3 steps until Candidate Remaining bids Discard Winning bid All bids Discard Winning bids

15
**Allocation Scheme (2) The candidate selection is greedy PU Gain**

The largest platform utility gain per unit price Platform utility gain: PU Gain Candidate Price Current distribution Updated distribution

16
**Allocation Scheme (3) Proportional rule check Soft label update**

Collect the answer from the winning bid Update the soft label according to Bayes rule budget price fraction ratio

17
**Allocation Scheme (5) Computationally efficient ! Candidate selection**

Proportional rule check Soft label update Computationally efficient !

18
**Payment Scheme(1) {A, B, C} {D, E, F} {A, B, D, E} {F}**

p(C) = max {b1,b2, b3, b4} Winning bids {A, B, C} Discard {D, E, F} Kick out C { A,B,D,E,F } Winning bids {A, B, D, E} Discard {F} b2 b3 b1 b4 b1 is the minimum price that bid C can replace bid A C

19
Payment Scheme(2) (Proposition)The winning bid C is paid threshold payment. p(C) C’s payment, b(C) C’s bid if b(C) < p(C), C is a winning bid if b(C) > p(C), C is discarded p(C)=max { b1, b2, b3, b4} Winning bids {A, B, D, E} b2 b3 b1 b4 C

20
**Payment Scheme(3) (Proposition)The incentive mechanism is truthful**

Each bid has a cost Workers will truthfully reveal the cost as asked price Why? Proof: Threshold payment + Greedy candidate Selection

21
**Individual Rationality**

(Proposition)The incentive mechanism is individual rational The utility of a winning bid is nonnegative Proof： Let us consider the winning bid C C is the 3rd winning bid. The first 2 bids are the same b3 is the minimum price that bid C can replace the new 3rd bid (D) It is true that b3 > b(c) ！ p(C) = max {b1, b2, b3, b4}, p(C) > b3 p(C) > b(C) New Winning bids {A, B, D, E} b2 b3 b1 b4 { A, B, C} Original wining bids

22
Budget Feasibility (Proposition, Payment Bound) Payment to each winning bid is upper bounded by Proportional rule: Set

23
**Performance Evaluation(1)**

Benchmark Untruthful Allocation: Workers’ cost is public information Random Allocation: Candidate selection is random Truthful Running Time Platform Utility Benchmark 1 High Benchmark 2 Low My Mechanism

24
**Performance Evaluation(2)**

Metric 1 : Platform Utility Platform utility vs. Budget Price of Truthfulness Gain over random allocation

25
**Performance Evaluation(3)**

Metric 2 : Budget Utilization Payment / Budget Budget utilization gain Over random allocation

26
Thank you ! Presented by : Qi Zhang

Similar presentations

OK

COMP 2208 Dr. Long Tran-Thanh University of Southampton Bandits.

COMP 2208 Dr. Long Tran-Thanh University of Southampton Bandits.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on earth dam failure Ppt on sources of energy for class 8th science Ppt on electric meter testing companies Ppt on polynomials free download Ppt on mammals and egg laying animals birds Ppt on railway track cross Ppt on shell and tube heat exchanger Ppt on conservation of wildlife and natural vegetation in california Ppt on phonetic transcription dictionary Ppt on understanding secularism in india