Presentation is loading. Please wait.

Presentation is loading. Please wait.

Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign

Similar presentations


Presentation on theme: "Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign"— Presentation transcript:

1 Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign X = $5000

2 Motivation Unlabeled data is free Labels are useful We need large volumes of labeled data Different labeling needs: Is there X in the image? Outline X. Where is part Y of X. Of these 500 images, which belong to category X? ……………. and many more ……………….

3 Task Amazon Mechanical Turk Is this a dog? o Yes o No Workers Answer: Yes Task: Dog? Pay: $0.01 Broker $0.01

4 Motivation X = $5000 Custom annotations Large scaleLow price

5 Annotation protocols Type keywords Select relevant images Click on landmarks Outline something Detect features ……….. anything else ………

6 Type keywords $0.01

7 Select examples Joint work with Tamara and Alex Berg

8 Select examples requester mtlabel $0.02

9 Click on landmarks $0.01

10 Outline something $0.01 Data from Ramanan NIPS06

11 Detect features Measuring molecules. Joint work with Rebecca Schulman (Caltech) ?? $0.1

12 Motivation X = $5000 Custom annotations Large scaleLow price

13 Issues Quality? –How good is it? –How to be sure? Price? –How to price it? How does MTurk compare with others? How do I sign up?

14 Annotation quality Agree within 5-10 pixels on 500x500 screen There are bad ones. ACEG

15 Grading tasks Take 10 submitted results Create new task to verify the result Verification is easy –Pay the same or slightly higher price Total overhead - 10% (work in progress)

16 Price $0.01 per image (16 clicks) ~ $1500 / images >1000 images per day <4 months Workers suggested $ $0.05/img –$ $5500 / images

17 Is the price right? $0.01/ 40 clicks 15 hours 900 labels $0.01 / 14 clicks 1.6 hours 900 labels $0.01 / 16 clicks 4 hours 900 labels

18 Annotation Method Comparison ApproachCostScaleSetup effort CentralizedQualityElastic to $ MTurk$+++*no+/ LabelME++Yes++ ImageParsing.com$$++**Yes Games with purpose (ESP++) ++++***Yes++ In house$$$+*no+++++

19 How do I sign up? Go to our web page: Send us an Register at Amazon Mechanical Turk

20 Acknowledgments Special thanks to: David Forsyth Tamara Berg Rebecca Schulman David Martin Kobus Barnard Mert Dikmen All workers at Amazon Mechanical Turk This work was funded in part by ONR

21 Thank you X = $5000


Download ppt "Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign"

Similar presentations


Ads by Google