Crowdsourcing Quality Management and other stories Panos Ipeirotis New York University & Tagasauris.

Slides:

Advertisements

Similar presentations

Quality Management on Amazon Mechanical Turk Panos Ipeirotis Foster Provost Jing Wang New York University.

Advertisements

______s Room Budget Created by:. My New Room I plan on purchasing _____ items for $ ____________.

Clustering Art & Learning the Semantics of Words and Pictures Manigantan Sethuraman.

The minnesota digital library coalition Achieving Digital Preservation Through Collaboration Museum Computer Network 32 nd Annual Meeting Keith Ewing St.

Crowdsourcing using Mechanical Turk Quality Management and Scalability Panos Ipeirotis – New York University Title Page.

Estimating the Completion Time of Crowdsourced Tasks using Survival Analysis Jing Wang, New York University Siamak Faridani, University of California,

Introduction to Mechanized Labor Marketplaces: Mechanical Turk Uichin Lee KAIST KSE.

Presenter: Chien-Ju Ho  Introduction to Amazon Mechanical Turk  Applications  Demographics and statistics  The value of using MTurk Repeated.

Presented to MCPS Teaching and Learning Committee January 6, 2014.

MEP Tutor Training January 2011

Crowdsourcing research data UMBC ebiquity,

Internet Enabled Human Computation CSE 454 Daniel Weld.

Crowdsourcing = Crowd + Outsourcing “soliciting solutions via open calls to large-scale communities”

Instructions: Your team should use the following slides for your camera point. You need to make 3 different sets of slides – one for each of your three.

String Theory The Theory of Everything Ryan Tiraschi.

Functional Skills Level 1 Maths revision

Human Computation and Crowdsourcing Uichin Lee May 8, 2011.

*Chapter One: What is Footnote?* Footnote allows people to find and share over 70 million historical documents Use the search engine to explore documents.

ESA2 January 2009.

Ohio Social Studies Standards Grade: 2 Lesson: Ohio Authors: Nathalie Wilmer & Jacob White.

Primary and Secondary Resources for Secondary Teachers August 16, 2012 Access “Media Talk”, our District Media Center wiki, using

The Western Waters Digital Library: Building a Resource Through Multi- State Collaboration and Technology Dawn Paschal Assistant Dean, Digital Library.

Spam? No, thanks! Panos Ipeirotis – New York University ProPublica, Apr 1 st 2010 (Disclaimer: No jokes included)

Crowdsourcing with Multi- Dimensional Trust Xiangyang Liu 1, He He 2, and John S. Baras 1 1 Institute for Systems Research and Department of Electrical.

Personal Information Management Vitor R. Carvalho : Personalized Information Retrieval Carnegie Mellon University February 8 th 2005.

Why Arizona: An Arizona Centennial Public Engagement Project Rob Spindler, Head of Archives & Special Collections Arizona State University Libraries.

Using local studies resources at Derbyshire libraries Cultural and Community Services.

Crowdsourcing using Mechanical Turk Quality Management and Scalability Panos Ipeirotis – New York University.

HIDDEN HISTORIES CAPTURING THE LAYERS OF CITIES BY MAPPING MEMORIES.

Organizing your desktop folders Learning Target: I can organize my desktop folder so that all of my do nows, digital photographs, formative projects and.

Photojournalism: the photo essay Dream Street: W. Eugene Smith’s Pittsburgh Project.

I NVESTIGATING Ashley Butler, Rebecka Embry, Jo Lammert INF385S Digital Libraries February 17, 2011.

Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.

Designing Your Dream Bedroom

Crowdsourcing using Mechanical Turk Quality Management and Scalability Panos Ipeirotis – New York University.

Online Public Access ACERA Spring Meeting April 6, 2011 Pamela Wright Chief Digital Access Strategist, NARA 1.

Welcome back! Mr. Schmidt 7A Math. My background I grew up in Los Angeles, CA. I attended Arizona State University for my Bachelor’s and Master’s degrees.

MULTIPLYING DECIMALS. NS 1.4 Calculate given percentages of quantities and solve problems involving discounts at sales, interests earned, and tips. Today’s.

Kinematics of a Rubber Bouncy Ball A Simulation of a Rubber Bouncy Ball Yun Chung Lu Physics 210 Final Project Presentation Affiliation: University of.

NOTES Multiplying Decimals MULTIPLYING DECIMALS: 1 st MULTIPLY AS NORMAL 2 nd Count the number of digits behind the decimal in the factors (in the problem)

Tiziana // Alessandra Lenzi - MG Breaking down the walls Project Museo Galileo and the Linked Open Data A joint project between.

1 Upper West Side/Central Park West Historic District Protective Mechanisms R8-B Contextual Zoning Districts W. 70 th Street midblock – north side Upper.

Data Handling Multiple Choice (BCD Questions). A B C D Q1. An ordinary fair dice is thrown at the same time a a coin is tossed. The probability of obtaining.

Sourcing CLOSE READ Before/During/After CROSS-CHECKING.

Memory Masters Preserving Digitized Histories— for today, for tomorrow, and for the future This project is made possible by a grant from the federal Institute.

A DΙgital Library Infrastructure on Grid EΝabled Technology SAPIR – Search in Audio Visual Content.

Crowdsourcing diligent search Maurizio Borghi Professor of Law Director, Centre for Intellectual Property Policy & Management Bournemouth University 3rd.

Employability Skills Portfolio material.aspwww.communitycc.ca/ccc/resources/prog material.asp.

How much does and should digitization cost? Nikki Ferraiolo (moderator) CLIR clir.org/hiddencollections Sandra McIntyre Mountain.

Entry Task May 2, 2016 Pick up a Graph Sheet as you come into the room. Be in your seats by the time the bell rings. While I Check Vocabulary: 1.Plot out.

Probability of Compound Events. 1. On the i-Pad, go to the Quizlet app 2.In the search screen type: Math 7 Chapter 9 probability 3.Click Math 7 Chapter.

MEGAN BROWN DPLA: DIGITAL PUBLIC LIBRARY OF AMERICA.

Crowdsourcing in Archives & Libraries

Omeka Web-Publishing Platform

The Right to Know An Overview of Library and Archives Canada’s Role in Truth and Reconciliation Johanna Smith Director General, Public Services Branch.

How do historians study communities, heroes, and helpers?

WJEC GCSE in Geography Place.

Scientific Investigation

The need for standardization

Primary vs. Secondary Sources

Using Publisher.

Images of the Great Depression ( )

How to do an INVESTIGATION

Warm Up Wed/Thurs. Do #17 p. 341 You will want this on a piece of paper to KEEP for reference.

A graphing calculator is required for some problems or parts of problems 2000.

“Teach A Level Maths” Statistics 1

U.S. National Parks Why have National Parks?

Presentation transcript:

Crowdsourcing Quality Management and other stories Panos Ipeirotis New York University & Tagasauris

Amazon Mechanical Turk

patent pending Tagging

patent pending But what if MTurk workers submit incorrect tags?

Voting

But what if MTurk workers also vote incorrectly?

Voting But what if MTurk workers also vote incorrectly? Redundancy: 5+ votes per tag

But my task is not just right or wrong… Creation HIT (e.g. describe the image) Improve HIT (e.g. improve description) Improve HIT (e.g. improve description) Compare HIT (voting) Which is better? Compare HIT (voting) Which is better? Describe this

version 1: A parial view of a pocket calculator together with some coins and a pen. version 2: A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocket and wallet sized item used for business, writting, calculating prices or solving math problems and purchasing items. version 3: A close-up photograph of the following items: A CASIO multi- function calculator. A ball point pen, uncapped. Various coins, apparently European, both copper and gold. Seems to be a theme illustration for a brochure or document cover treating finance, probably personal finance. version 4: …Various British coins; two of £1 value, three of 20p value and one of 1p value. … version 8: “A close-up photograph of the following items: A CASIO multi-function, solar powered scientific calculator. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of £1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance."

New York: Old and New An engagement experiment: “Search the NYPL digital library for an old photo of New York, and then find a photo of the same location, today”

1935

2007

1933

2007

1890

2007

1890

1900

1976

2008

The incomplete knowledge graph [Harsen Homestead, 72nd Street, ca ]

The incomplete knowledge graph Harsen Homestead ???

The incomplete knowledge graph Jacob Harsen, a farmer who started Harsenville, a village spanning 68th Street to 81st Street between Central Park West and the Hudson River. Source:

The incomplete knowledge graph Crowd not only annotates historical archives but also contributes back to the knowledge graph [allowing easier annotation in the future]

New York Times, 1911

The incomplete knowledge graph Crowd participation not only annotates historical archives but also contributes back to the knowledge graph [allowing easier annotation in the future]