Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.

Slides:

Advertisements

Similar presentations

CrowdER - Crowdsourcing Entity Resolution

Advertisements

Crowdsourcing ontology engineering Elena Simperl Web and Internet Science, University of Southampton 11 April 2013.

Low Cost, Scalable Proteomics Data Analysis Using Amazon's Cloud Computing Services and Open Source Search Algorithms Brian D. Halligan, Ph.D. Medical.

Estimating the Completion Time of Crowdsourced Tasks using Survival Analysis Jing Wang, New York University Siamak Faridani, University of California,

MT Evaluation: Human Measures and Assessment Methods : Machine Translation Alon Lavie February 23, 2011.

Bringing the crowdsourcing revolution to research in communication disorders Tara McAllister Byun, PhD, CCC-SLP Suzanne M. Adlof, PhD Michelle W. Moore,

Wednesday, February 24th Agenda

Gold Sponsors Bronze Sponsors Silver Sponsors Building a SharePoint Forms Platform: Real World Considerations and Lessons Martin Harris OBS.

Running Experiments with Amazon Mechanical-Turk Gabriele Paolacci, Jesse Chandler, Jesse Chandler Judgment and Decision Making, Vol. 5, No. 5, August 2010.

Try again, er something. February 17, 2005 Greg Giacovelli – Nick Mancuso – Shaun Newsum – Jean-Paul Pietraru – Nick Stroh.

Crowdsourcing research data UMBC ebiquity,

CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin.

Virtual techdays INDIA │ 9-11 February 2011 No Code SharePoint Development using SharePoint Designer 2010 Roshan N.Y │ Consultant, Microsoft.

M.A.Doman Model for enabling the delivery of computing as a SERVICE.

Microsoft Kinect TM + Crowdsourcing A Prototype for Code Validation Matt Trippy, Owner Enovation Concepts.

WORKFLOWS IN CLOUD COMPUTING. CLOUD COMPUTING  Delivering applications or services in on-demand environment  Hundreds of thousands of users / applications.

From Visions to Reality Road Show II Mexico in European Union.

Presenters: Title:. CONTENTS What is Crowdsourcing? How Crowdsourcing works? Types of Crowdsourcing Applications of Crowdsourcing Benefits & Problems.

Crowdsourcing Quality Management and other stories Panos Ipeirotis New York University & Tagasauris.

NEHAL PATEL GMO OCTOBER 21, 2008 ProActive for Algorithmic Trading.

Students: Ilya Paskhover, Itay Gal Supervisors: Oleg Rokhlenko, Nadav Golbandi.

Human Computation and Crowdsourcing Uichin Lee May 8, 2011.

1 © Copyright 2012 EMC Corporation. All rights reserved. EMC VSPEX Simple. Efficient. Flexible.

The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.

CMS Human Resources Labor Cost Distribution 8.9 Training.

Teaching End User SharePoint Robert Bogue

Christopher Harris Informatics Program The University of Iowa Workshop on Crowdsourcing for Search and Data Mining (CSDM 2011) Hong Kong, Feb. 9, 2011.

DATA-CENTERED CROWDSOURCING WORKSHOP PROF. TOVA MILO SLAVA NOVGORODOV TEL AVIV UNIVERSITY 2014/2015.

RootsTech 2012: My Experiences Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community

Science & Technology Centers Program Center for Science of Information National Science Foundation Science & Technology Centers Program Bryn Mawr Howard.

Varun Sharma Application Consulting and Engineering (ACE) Team, Microsoft India.

Crowdsourcing: Ethics, Collaboration, Creativity KSE 801 Uichin Lee.

The Strategy Unit Plan. Begin by noting the name of the writing strategy students will learn PLANS essay strategy.

Department of Informatics, UC Irvine SDCL Collaboration Laboratory Software Design and sdcl.ics.uci.edu 1 Search and the Crowd: Next-Generation Software.

The future of crowd work Aniket Kittur et al., (2013). The future of crowd work, CSCW’13 Mejdl Safran Fall 2013 SIUC.

Crowdsourcing Using SIS Duncan Yung. Crowdsourcing Example: Find the lowest price of a product 1.Distribute the task to different workers. 2.Workers submit.

Presented by D’Sunte Wilson Microsoft Dynamics Nav (Navision) Training – Day 2.

Crowdsourcing using Mechanical Turk Quality Management and Scalability Panos Ipeirotis – New York University.

The Rise of Crowdsourcing in Management Research Organized by: Yuqing (Ching) Ren Natalia Levina August 9, 2010 Academy of Management Annual Meeting.

© 2009 Amazon.com, Inc. or its Affiliates. Insights into Mechanical Turk (or, “Mistakes Requesters Make”) Adam D. Bradley

Consensus Relevance with Topic and Worker Conditional Models Paul N. Bennett, Microsoft Research Joint with Ece Kamar, Microsoft Research Gabriella Kazai,

CrowdForge: Crowdsourcing Complex Work Aniket Kittur, Boris Smus, Robert E. Kraut February 1, 2011 Presenter: Karalis Alexandros 1.

Flowcharts C++ Lab. Algorithm An informal definition of an algorithm is: a step-by-step method for solving a problem or doing a task. Input data A step-by-step.

Crowdsourcing Blog Track Top News Judgments at TREC Richard McCreadie, Craig Macdonald, Iadh Ounis {richardm, craigm, 1.

American History Unit: World War I A Day: Monday, February 1, 2016 B Day: Tuesday, February 2, 2016.

Crowd-based mining of reusable process model patterns Carlos Rodríguez, Florian Daniel, Fabio Casati BPM 2014, September 9th 2014, Eindhoven, The Netherlands.

This document and the information contained herein is confidential and proprietary to Allegient LLC and shall not be duplicated, used or disclosed in whole.

Designing a Human-Machine Hybrid Computing System for Unstructured Data Analytics KOUSHIK SINHA, GEETHA MAJUNATH, BIDYUT GUPTA, SHAHRAM RAHIMI.

Bossa: A platform for distributed thinking David P. Anderson UC Berkeley Space Sciences Lab 11 Oct 2007.

Unit 1 Days 1-2: Standard 1 Objective 1

Data-Centered Crowdsourcing Workshop

The platform economy Impact on employment and social security in the EU Prof. Martin Risak Department for Labour Law and Law of Social Security, University.

Working with Workflow and Forms (Lab)

Crowdsourcing: A New Work Style

SB 1843 – Armed services vocational aptitude battery (Asvab) test

for Situationally Appropriate Interactions

Implementing AI solutions using the cognitive services in Azure

Emerging Frontiers of Science of Information

oTree: An open-source platform for lab, web, and field experiments

Brad Zdenek Innovation Strategist January 2018

Grammar Starter Fewer or less? All ✰ Most ✰✰ Some ✰✰✰

“The Golden Kite, The Silver Wind”

What is this and how can I use it?

What is this and how can I use it?

What is this and how can I use it?

HR Specialty Products & Services Catalogue Executive Summary

Mechanical Turk Lessons: CardLasso

Curitiba: cities and self organisation

SB 1843 – Armed services vocational aptitude battery (Asvab) test

Presentation transcript:

Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013

Agenda Crowdsourcing History and Background Crowdsourcing Demo Worker Quality Control Theories and Tactics Crowdsourcing War Stories

Isaac’s Background Former UW/CSE Student. Worked for Microsoft & then Amazon after graduation. Responsible for launching Amazon’s Mechanical Turk on Nov 2005 Sold crowdsourcing consulting business, TurkForce, to CrowdFlower in 2010 Founded Chirpsy in 2011…

Nathan’s Background Started Career by working for 7 startups in 6 years, doing all manner code. Followed that with a stint at Los Almos National Labs - doing a bunch of more academic things. Founded CastingWords in Nov 2005 right when MTurk launched, ~7 year ago.

What is crowdsourcing? Crowdsourcing is … “Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

Crowdsourcing is … “Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

What is crowdsourcing? Crowdsourcing is … “Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

What is crowdsourcing? Crowdsourcing is … “Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

Crowd Labor Definition: Leveraging of a distributed virtual labor pool, available on-demand to fulfill a range of tasks from simple to complex Simple Micro-task Crowdsourcing Complex Task Crowdsourcing

History Lesson: The Original Mechanical Turk

5 Minute Mechanical Turk Demo…

Worker Quality Control Plurality / Majority Rules Gold Standard Questions / Known Answers Peer Review / Grader Validation Workflows Algorithmic Approaches (Machine Learning) Worker Reputation Submission Credibility (Dynamic Quality Control)

Crowdsourcing War Stories The Good… –Workers can be extremely helpful and proactive The Bad... –Mistakes can have long term consequences The Ugly… –Unpredictable Fraud

Questions?