Presentation is loading. Please wait.

Presentation is loading. Please wait.

Crowdsourcing with Amazon Mechanical Turk

Published byKelley Griffin Modified over 8 years ago

Similar presentations

Presentation on theme: "Crowdsourcing with Amazon Mechanical Turk"— Presentation transcript:

1 Crowdsourcing with Amazon Mechanical Turk
Alex Quinn and Tom Yeh HCIL Symposium May 25, 2011 slight correction (crowdflow -> crowdlib) on 6/6/2011.

2 Interface options Easy web-based tool REST and SOAP APIs (HTTP)
REST and SOAP APIs (HTTP) Wrapper SDKs: Java, .NET, Perl, PHP, Ruby Command line tool TurKit (JavaScript) Boto (Python) CrowdLib (Python)

3 Posting HITs – external
Your URL in an IFRAME Your form must route results to Amazon. <form method="post" action="

4 Posting HITs – standard
QuestionForm XML

5 Standard vs. External HITs
Amazon hosts everything Content: HTML, Flash, applets Answers: Standard controls No JavaScript / CSS More secure Easier to get started Requires your own web server Content: anything! Answers: Your CGI form fields Inner scrolling (potentially) More flexible Easier to do fancy things

6 API Documentation

7 Question XML <QuestionForm xmlns="..."> <Question> <QuestionIdentifier>food_name</QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>What food is Maine known for?</Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer></FreeTextAnswer> </AnswerSpecification> </Question> </QuestionForm>

8 Example - TurKit var params = { title : "Find food for a state", desc : "Given a US state find a food it is known for", question : questionXML, reward : 0.01, maxAssignments : 3 }; var hit = mturk.createHIT(params); hit = mturk.getHIT(hit,true); for( var j=0; j<hit.assignments.length; j++ ) { var foodName = hit.assignments[j].answer.food_name; print(stateName + " : " + foodName); } <QuestionForm xmlns="..."> <Question> <QuestionIdentifier>food_name</QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>What food is Maine known for?</Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer></FreeTextAnswer> </AnswerSpecification> </Question> </QuestionForm>

9 Example - CrowdLib import crowdlib as cl, crowdlib_settings hit_type = cl.create_hit_type("Find food for a state", "Given a US state find a food it is known for", 0.01) fields = [cl.text_field("What food is Maine known for?")] hit = hit_type.create_hit(fields, max_assignments=3)

10 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results

11 Example – Command Line Many little files! Creates even more…
state_foods.question  question XML state_foods.input  parameters (i.e. states) state_foods.properties  title, description, etc. state_foods_start.bat  calls loadHITs.bat state_foods_get_results.bat  fetch results state_foods_review_results.bat  approve/reject Creates even more… state_foods.success  contains the HIT IDs state_foods.results  all result details

12 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results <QuestionForm xmlns="..."> <Question> <QuestionIdentifier> food_name </QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text> What food is Maine known for? </Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer/> </AnswerSpecification> </Question> </QuestionForm>

13 Example – Command Line Many little files! Creates even more…
state Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine ... Many little files! state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results

14 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results title:Find food for a state description:Given a US state find a food it is known for keywords:culture, food, search reward:0.02 assignments:1 annotation:${state} # time limit: 30 minutes assignmentduration:1800 # expires in: 3 days hitlifetime:259200 # auto-pay after: 2 days autoapprovaldelay:

15 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results pushd bin call loadHITs %1 %2 %3 %4 %5 ^ -label ..\sf\state_foods ^ -input ..\sf\state_foods.input ^ -question ..\sf\state_foods.question ^ -properties ..\sf\state_foods.properties popd

16 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results pushd ..\bin call getResults %1 %2 %3 %4 %5 ^ -successfile ..\sf\state_foods.success -outputfile ..\sf\state_foods.results popd

17 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results pushd ..\..\bin call reviewResults %1 %2 %3 %4 %5 ^ -resultsfile ..\sf\state_foods.results popd

18 Example – Command Line Many little files! Creates even more…
state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results hitid hittypeid 11MFAR2I9GUBPPX3H302KECWDGG45C 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1N2WE274D6AYYDS5F6ENYRH1NR7JAO 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1KBVYNQOT8Y85Z3T3H6YTS1CV3596T 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1NPDB0OPE4D7JGXHYWSMAPKKCVAELG 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1HNBWMEOVNP9NL2O6SYS8WM19JGNIZ 1MQHAMAJ67VR562077Y5NLM6WNOIO8

19 Example – Command Line Many little files! Creates even more…
"hitid" "hittypeid" "title" "description" "keywords" "reward" "creationtime" "assignments" "numavailable" "numpending" "numcomplete" "hitstatus" "reviewstatus" "annotation" "assignmentduration" "autoapprovaldelay" "hitlifetime" "viewhit" "assignmentid" "workerid" "assignmentstatus" "autoapprovaltime" "assignmentaccepttime" "assignmentsubmittime" "assignmentapprovaltime" "assignmentrejecttime" "deadline" "feedback" "reject" "Answer.answer" "11MFAR2I9GUBPPX3H302KECWDGG45C" "1MQHAMAJ67VR562077Y5NLM6WNOIO8" "Find food for a state" "Given a US state find a food it is known for" "US culture, food, search" "$0.02" "Mon Dec 06 21:50:25 EST 2010" "1" "0" "0" "0" "Reviewable" "NotReviewed" "Maine" "3600" " " "Thu Dec 09 21:50:25 EST 2010" " HIT?HITId=11MFAR2I9GUBPPX3H302KECWDGG45C" "1SPKY563G67J32JCIGFCFF0CVAKGF0" "A1JYO3377GTS1O" "Submitted" "Tue Dec 21 21:52:56 EST 2010" "Mon Dec 06 21:51:26 EST 2010" "Mon Dec 06 21:52:56 EST 2010" "" "" "" "" "" "lobster" Many little files! state_foods.question state_foods.input state_foods.properties state_foods_start.bat state_foods_get_results.bat state_foods_review_results.bat Creates even more… state_foods.success state_foods.results

20 Reviewing assignments
Approve Reject Grant bonus Word to wise: Preview in Excel!

21 Worker issues Approval rating
Blocked by multiple requestors  banned!!! Too many rejections  banned (???) My rule: If they made a reasonable attempt to follow the directions, then pay. If quality is bad, get in touch.

22 Worker issues Indiscriminate rejections…

23 Qualification Requirements
Qualification Type ~ HIT Type Qualification Requirement ~ HIT Qualification ~ Assignment Custom Built-in Granted

24 Informed Consent

25 Ways to receive results
Ad-hoc script Polling Notification receptor

26 Terminology review HIT Type HIT HIT Type HIT
Assignment  (HIT, worker) Answer  part of Assignment Qualification Type Qualification Requirement Qualification  (QR, worker) HIT Type HIT

Download ppt "Crowdsourcing with Amazon Mechanical Turk"

Similar presentations

Trends in Number of High School Graduates: National

Trends in Number of High School Graduates: National
The West` Washington Idaho 1 Montana Oregon California 3 4 Nevada Utah

The West` Washington Idaho 1 Montana Oregon California 3 4 Nevada Utah
Exhibit 1. Premiums and Premium Increases, Single Coverage, by Metal Tier and State, 2014–2015 Premium increases, 2014–20152015 premiums for 40-year-old.

Exhibit 1. Premiums and Premium Increases, Single Coverage, by Metal Tier and State, 2014–2015 Premium increases, 2014– premiums for 40-year-old.
Being Money Wise. Question Why do you have to be wise with your money?

Being Money Wise. Question Why do you have to be wise with your money?
TOTAL CASES FILED IN MAINE PER 1,000 POPULATION CALENDAR YEARS 1999 - 2009 FILINGS PER 1,000 POPULATION This chart shows bankruptcy filings relative to.

TOTAL CASES FILED IN MAINE PER 1,000 POPULATION CALENDAR YEARS FILINGS PER 1,000 POPULATION This chart shows bankruptcy filings relative to.
By Sam Kider Miles Kvalheim Matt Varghese.  Do states that spend more money on education have less crime?

By Sam Kider Miles Kvalheim Matt Varghese.  Do states that spend more money on education have less crime?
BINARY CODING. Alabama Arizona California Connecticut Florida Hawaii Illinois Iowa Kentucky Maine Massachusetts Minnesota Missouri 0 Nebraska New Hampshire.

BINARY CODING. Alabama Arizona California Connecticut Florida Hawaii Illinois Iowa Kentucky Maine Massachusetts Minnesota Missouri 0 Nebraska New Hampshire.
This chart compares the percentage of cases filed in Maine under chapter 13 with the national average between 1999 and 2009. As a percent of total filings,

This chart compares the percentage of cases filed in Maine under chapter 13 with the national average between 1999 and As a percent of total filings,
Map Review. California Kentucky Alabama.

Map Review. California Kentucky Alabama.
Judicial Circuits. If You Live In This State This Is Your Judicial Circuit Alabama11th Circuit Alaska 9th Circuit Arkansas 8th Circuit Arizona 9th Circuit.

Judicial Circuits. If You Live In This State This Is Your Judicial Circuit Alabama11th Circuit Alaska 9th Circuit Arkansas 8th Circuit Arizona 9th Circuit.
1. AFL-CIO What percentage of the funds received by Alabama K-12 public schools in school year 2010-2011 was provided by the state of Alabama? a)44% b)53%

1. AFL-CIO What percentage of the funds received by Alabama K-12 public schools in school year was provided by the state of Alabama? a)44% b)53%
The United States.

The United States.
Directions: Label Texas, Arkansas, Louisiana, Mississippi, Tennessee, Alabama, Georgia, Florida, South Carolina, North Carolina, Virginia--- then color.

Directions: Label Texas, Arkansas, Louisiana, Mississippi, Tennessee, Alabama, Georgia, Florida, South Carolina, North Carolina, Virginia--- then color.
 As a group, we thought it be interesting to see how many of our peers drop out of school.  Since in the United States education is so important, we.

 As a group, we thought it be interesting to see how many of our peers drop out of school.  Since in the United States education is so important, we.
CHAPTER 7 FILINGS IN MAINE CALENDAR YEARS 1999 – 2009 CALENDAR YEAR CHAPTER 7 FILINGS This chart shows total case filings in Maine for calendar years 1999.

CHAPTER 7 FILINGS IN MAINE CALENDAR YEARS 1999 – 2009 CALENDAR YEAR CHAPTER 7 FILINGS This chart shows total case filings in Maine for calendar years 1999.
©SHRM 2008SHRM Poll: May 20, 20081 Number of Out-of-Town Job Candidates (from Distant Cities, States) in the Last 12 Months Compared With Previous Years.

©SHRM 2008SHRM Poll: May 20, Number of Out-of-Town Job Candidates (from Distant Cities, States) in the Last 12 Months Compared With Previous Years.
Study Cards The East (12) Study Cards The East (12) New Hampshire New York Massachusetts Delaware Connecticut New Jersey Rhode Island Rhode Island Maryland.

Study Cards The East (12) Study Cards The East (12) New Hampshire New York Massachusetts Delaware Connecticut New Jersey Rhode Island Rhode Island Maryland.
BY NATALIE CZECH 3 rd Grade Social Studies. Standards and Objectives V. GEOGRAPHY B. Maps and Globes  The student will use maps and globes to demonstrate.

BY NATALIE CZECH 3 rd Grade Social Studies. Standards and Objectives V. GEOGRAPHY B. Maps and Globes  The student will use maps and globes to demonstrate.
Hawaii Alaska (not to scale) Alaska GeoCurrents Customizable Base Map text.

Hawaii Alaska (not to scale) Alaska GeoCurrents Customizable Base Map text.
US MAP TEST Practice. 1 6 4 9 8 7 12 11 10 5 2 13 14 15 21 20 19 18 1716 3 22 31 30 29 28 27 26 25 24 23 40 39 38 37 36 35 34 33 32 45 44 43 48 42 41.

US MAP TEST Practice

Similar presentations

Ads by Google