1. Pricing and Incentives I KSE 801 Uichin Lee

2. Financial Incentives and the Performance of Crowds Winter Mason and Duncan J. Watts KDD-HCOMP 09, June 28, 2009 Paris, France

3. Motivation of Crowdsourcing: Monetary incentive vs. Performance Motivation of crowdsourcing and peer production – Intrinsic motivation: enjoyment, desire to help out – Social rewards (Flickr) – Monetary incentives (M-Turk) Rational choice in economic theory – Rational workers will choose to improve their performance in response to a scheme that rewards such improvements with financial gain Example: an autoglass factory where workers install windshields on a production line and switched from time- rate wage (pay per hour) to piece-rate wage (pay per unit) – Over the course of a year and half, individual productivity has increased 20%, concluding that performance based pay scheme is a powerful tool for eliciting improved performance

4. Motivation of Crowdsourcing: Monetary incentive vs. Performance Yet, under certain circumstance the provision of financial incentives can undermine intrinsic motivation (e.g., enjoyment, altruism), possibly leading to poorer outcome Workers may ignore rational incentives to work longer when they have accomplished pre-set targets (e.g., taxi driver examples) Recent experiments demonstrate that financial incentives undermine actual performance (called chocking effect); e.g., hampering innovations

5. Experiments Study 1: Image ordering – Image ordering: sorting a set of images taken from a traffic camera at 2-second intervals in chronological order – To understand the impact of monetary compensation on performance (quantity and quality) – Differentiate between quantity of work (output) and quality of work (accuracy) – Use image ordering tasks where output can vary widely and accuracy can be measured objectively Study 2: Word puzzles – To further investigate the absence of an effect on accuracy (will see what it means later)

6. Study 1: Image ordering At the beginning of a task, you will be presented with a list of images taken from traffic cameras. An example list is shown below Your goal is to reorder the list chronologically from left to right and top to bottom. The sorted list is shown below. Drag and drop for ordering

7. Study 1: Image ordering All participants (total 611 workers) were paid an initial fee of $0.10 to complete the introductory survey and training set Then, participants were given information about (1) how much they would be paid (pay level) and (2) what difficulty level they will be given – Pay level: No pay Low pay: $0.01 per image set Medium pay: $0.05 per image set High pay: $0.10 per image set – Difficulty level: Easy (two images per set) Medium (three images per set) Hard (four images per set)

8. Study 1: Image ordering – Results Across all difficulty levels participants chose to complete more tasks on average when the pay was higher – F(3, 607) = 15.73, p<0.001 Across all payment levels, the number of completed tasks decreased with increasing difficulty Number of tasks completed Pay per task Hard: 4 images Medium: 3 images Easy: 2images

9. Study 1: Image ordering – Results Accuracy measure: – A: Proportion of image sets correctly sorted – B: Spearmans rank correction between the correct order and the sorted order Perfect match: ρ = 1 Perfect mismatch: ρ = 0 One-way ANOVA test confirms that increasing wage did not improve accuracy – A: F(3, 607) = 0.66, p<0.001 – B: F(3, 607) = 0.82, p<0.001 Accuracy (proportion of images sets correctly labeled) Pay per task Hard: 4 images Medium: 3 images Easy: 2 images

10. Discussion Another test: single wage of $0.01, yet alerting participants that accuracy is measured, with the following 4 variants 1)Only informed that accuracy would be measured 2)Shown feedback on their accuracy after every fourth image set 3)Told explicitly that their pay would be contingent on their performance 4)(2) and (3) are used in combination With this test, they found that quantity and quality results were indistinguishable, suggesting that workers were treating their pay as performance dependent

11. Discussion: Anchoring effect? Workers in all conditions generally felt that the expected wage for the work they performed is greater than what they received – They felt they are paid less than expected – X 2 =243.61, p<0.001 Thus, workers were no more motivated to perform better no matter how much they were actually paid Perceived Task Value $0.10$0.01 $0.05 $0.20 $0.10 Actual Task Value $0.05

12. Study 2: Word Puzzle Want to further investigate absence of an effect on accuracy * Shown a list of 15 possible words (not all of the words listed are in the puzzle) * Select a word: click the first and last letter (if correct, it will turn red) * Shown a list of 15 possible words (not all of the words listed are in the puzzle) * Select a word: click the first and last letter (if correct, it will turn red) * Two wage models: quota vs. piece rate * Quota: every puzzle successfully completed * Piece: every word they found * Pay levels: low, medium, high, (no pay) -- Puzzle: $0.01, $0.05, $0.10 -- Word: $0.01, $0.02, $0.03 * Two wage models: quota vs. piece rate * Quota: every puzzle successfully completed * Piece: every word they found * Pay levels: low, medium, high, (no pay) -- Puzzle: $0.01, $0.05, $0.10 -- Word: $0.01, $0.02, $0.03

13. Study 2: Word Puzzle – Results Payment results in higher # of completions Incentive Framing Number of Puzzles Completed

14. Study 2: Word Puzzle – Results # of puzzles completed did not differ by payment strategies --- no significant impact on quantity! Pay per Puzzle Number of Puzzles Completed Pay per Word Number of Puzzles Completed F(2, 108) = 0.71, not significantF(2, 124) = 1.82, not significant 66 Possible Reason: Intrinsic motivation of enjoyment

15. Study 2: Word Puzzle – Results Accuracy (fraction of worlds found per puzzle) Cost per Word No Contingent PayPay per PuzzlePay per Word Accuracy Cost per word High accuracy per puzzle means low cost per word Low accuracy per puzzle, but workers find as many words as they can Intrinsic motivation (enjoyment)

16. Discussion Quota requires more efforts than piece rate Quota elicits more work: (1) greater marginal difficulties of finding next words to complete a puzzle, (2) implicit goal setting – harder puzzles become more salient goals Perceived Task Value Actual Task Value (Per Puzzle)Actual Task Value (Per Word) Anchoring effects

17. Discussion When there is no expectation of financial reward, effort is motivated by other kinds of rewards (e.g., social); but when monetary compensation is expected, as in the AMT framework, the anticipated financial value of the effort will be the driving mechanism. Although paid workers generally did more work than unpaid workers, how they were paid had a larger impact on their output and accuracy than how much they were paid. Moreover, paying workers a low rate led to them to perceive their work as less valuable than not paying them at all (previous slide)

18. Summary Investigated the relationship between financial incentives and performance in the novel setting of online peer production systems Main findings: – Increased payments increased the quantity of work performed, but not its quality – The particular design of the compensation scheme (a quota scheme vs. a piece rate, for example) can have a significant effect on quality even to the point where better work can be accomplished for less pay (or even without any payment)