1. Utility data annotation via Amazon Mechanical Turk Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign http://visionpc.cs.uiuc.edu/~largescale/ X 100 000 = $5000

2. Motivation Unlabeled data is free Labels are useful We need large volumes of labeled data Different labeling needs: Is there X in the image? Outline X. Where is part Y of X. Of these 500 images, which belong to category X? ……………. and many more ……………….

3. Task Amazon Mechanical Turk Is this a dog? o Yes o No Workers Answer: Yes Task: Dog? Pay: $0.01 Broker www.mturk.com $0.01

4. Motivation X 100 000 = $5000 Custom annotations Large scaleLow price

5. Annotation protocols Type keywords Select relevant images Click on landmarks Outline something Detect features ……….. anything else ………

6. Type keywords http://austinsmoke.com/turk/http://austinsmoke.com/turk/. $0.01

7. Select examples Joint work with Tamara and Alex Berg http://visionpc.cs.uiuc.edu/~largescale/data/simpleevaluation/html/horse.html

8. Select examples requester mtlabel $0.02

9. Click on landmarks $0.01 http://vision-app1.cs.uiuc.edu/mt/results/people14-batch11/p7/

10. Outline something $0.01 http://visionpc.cs.uiuc.edu/~largescale/results/production-3-2/results_page_013.html Data from Ramanan NIPS06

11. Detect features http://visionpc.cs.uiuc.edu/~largescale/all_examples.html Measuring molecules. Joint work with Rebecca Schulman (Caltech) ?? $0.1

12. Motivation X 100 000 = $5000 Custom annotations Large scaleLow price

13. Issues Quality? –How good is it? –How to be sure? Price? –How to price it? How does MTurk compare with others? How do I sign up? sorokin2@uiuc.edu http://visionpc.cs.uiuc.edu/~largescale/

14. Annotation quality Agree within 5-10 pixels on 500x500 screen There are bad ones. ACEG

15. Grading tasks Take 10 submitted results Create new task to verify the result Verification is easy –Pay the same or slightly higher price Total overhead - 10% (work in progress) http://vision-app1.cs.uiuc.edu/mt/grading/people14-batch11-small/p1/

16. Price $0.01 per image (16 clicks) ~ $1500 / 100 000 images >1000 images per day <4 months Workers suggested $0.03 - $0.05/img –$3500 - $5500 / 100 000 images

17. Is the price right? $0.01/ 40 clicks 15 hours 900 labels $0.01 / 14 clicks 1.6 hours 900 labels $0.01 / 16 clicks 4 hours 900 labels

18. Annotation Method Comparison ApproachCostScaleSetup effort CentralizedQualityElastic to $ MTurk$+++*no+/++++++++ LabelME++Yes++ ImageParsing.com$$++**Yes+++++++ Games with purpose (ESP++) ++++***Yes++ In house$$$+*no+++++

19. How do I sign up? Go to our web page: http://visionpc.cs.uiuc.edu/~largescale/ Send us an e-mail: sorokin2@uiuc.edu Register at Amazon Mechanical Turk http://www.mturk.com

20. Acknowledgments Special thanks to: David Forsyth Tamara Berg Rebecca Schulman David Martin Kobus Barnard Mert Dikmen All workers at Amazon Mechanical Turk This work was funded in part by ONR

21. Thank you X 100 000 = $5000