1. Introduction to Mechanized Labor Marketplaces: Mechanical Turk Uichin Lee KAIST KSE

2. Mechanical Turk Begin with a project – Define the goals and key components of your project. For example, your goal might be to clean your business listing database so that you have accurate information for consumers. Break it into tasks and design your HIT – Break the project into individual tasks; e.g., if you have 1,000 listings to verify, each listing would be an individual task. – Next, design your Human Intelligence Tasks (HITs) by writing crisp and clear instructions, identifying the specific outputs/inputs desired and how much you will pay to have work completed. Publish HITs to the marketplace – You can load millions of HITs into the marketplace. Each HIT can have multiple assignments so that different Workers can provide answers to the same set of questions and you can compare the results to form an agreed-upon answer. https://requester.mturk.com/tour/how_it_works

3. Mechanical Turk Workers accept assignments – If Workers need special skills to complete your tasks, you can require that they pass a Qualification test before they are allowed to work on your HITs. – You can also require other Qualifications such as the location of a Worker or that they have completed a minimum number of HITs. Workers submit assignments for review – When a Worker completes your HIT, he or she submits an assignment for you to review. Approve or reject assignments – When your work items have been completed, you can review the results and approve or reject them. You pay only for approved work. Complete your project – Congratulations! Your project has been completed and your Workers have been paid. https://requester.mturk.com/tour/how_it_works

4. Screenshot

6. AMT Questions Who are the workers that complete these tasks? What type of tasks can be completed in the marketplace? How much does it cost? How fast can I get results back? How big is the AMT marketplace? Analyzing the Amazon Mechanical Turk marketplace, P. G. Ipeirotis, Journal XRDS: Crossroads, 2010 Demographics of Mechanical Turk, P.G. Ipeirotis, NYU TR, 2010

7. Gender and Age Countries: 46.80% US, India: 34%, Misc: 19.2% (from 66 different countries) http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html

8. Educational Level Many of the workers are younger than overall population, and this leads to higher educational levels

9. Income Level Indian Turkers relatively have low income level

10. Marital Status and Household Size Lots of single workers Indian workers tend to belong to larger households

11. Level of Engagement on M-Turk Most workers spent less than a day per week, completing 20- 100 HITs, and earning less than $20 per week.

12. Motivation Why do you complete tasks in Mechanical Turk? Please check any of the following that applies (multiple items possible): – [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV) – [2] I find the tasks to be fun – [3] To kill time – [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards) – [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out) – [6] I am currently unemployed, or have only a part time job [1][2][3]

13. Motivation Why do you complete tasks in Mechanical Turk? Please check any of the following that applies: – [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV) – [2] I find the tasks to be fun – [3] To kill time – [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards) – [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out) – [6] I am currently unemployed, or have only a part time job [4][5][6]

14. Summary Significant fraction of Turkers are from US and India (> 80% vs. 20% from misc) Turkers are younger (more than 50% from 21-35) More females (US) and more males (India) Turkers relatively have lower income Turkers relatively have smaller families Geographic distribution of Turkers and Internet users is similar

15. Type of Tasks in M-Turk

16. Requester vs. Total Rewards Long tail nature of participation

17. Keywords vs. Ranks

18. Price Distribution HITgroups with a large number of HITs tend to have a low price (e.g., $0.10) 90% of HITs pay less than $0.10

19. Posting and Serving Process Cumulative distribution plots: – For each day, the value of tasks being posted by the AMT requesters, and the value of the tasks that got completed in each day Median posting/completion rate: $1,040 vs. $1,155 per day M/M/1 queueing assumption: a task worth $1, average completion time of 12.5 minutes Value of posted/completed HITs in USD ($)

20. Posting and Serving Process Less posting over the weekends (requesters) Less work done on Monday due to less posting over the weekends PostingCompletion

21. Completion Time Distribution 10 days Power law distribution: w/ alpha = -1.48

22. Next Week Taskcn (in China) How to use M-Turk for research? M-Turk test assignment