1. Investigating Factors Influencing Crowdsourcing Tasks with High Imaginative Load Raynor Vliegendhart Martha Larson Christoph Kofler Carsten Eickhoff (speaker) Johan Pouwelse WSDM 2011 Workshop on Crowdsourcing for Search and Data Mining (CSDM 2011), Hong Kong, China, February 9–12, 2011.

2. 2 O First Things First

3. 3 That title sounds pretty esoteric. What is this all about? O First Things First

4. 4 That title sounds pretty esoteric. What is this all about? We are dealing with two phenomena: HIT titles that try to prepare the worker for the task HITs that require the worker to project into different roles or situation We refer to this property of HITs as “Imaginative Load” O First Things First

5. 5 Why “Imaginative Load”? Observations Further Investigation Conclusions O Outline

6. 6 Evaluation context: Novel search-related feature for a file-sharing system Term clouds as content descriptors Required the workers to project themselves into the role of a user I Why “Imaginative Load”?

7. 7 The actual HIT was preceded by a recruitment step: R Why “Imaginative Load”? Recruitment HIT 100/100 Evaluation HIT 0/405 Qualified Workers 81 81

8. 8 The turnout: R Why “Imaginative Load”? Recruitment HIT 100/100 Evaluation HIT 10/405 4 Qualified Workers 81

9. 9 Perhaps we need more eligible workers? R Why “Imaginative Load”? Recruitment HIT 100/100 Evaluation HIT 12/405 5 Recruitment HIT 100/100 79 Qualified Workers 160

10. 10 HIT uptake is slow Most workers do not do the actual HIT If they do, then they don't do many iterations Hypothesis: “The recruitment task and the HIT titles were misleading. Once the workers realised what they were supposed to do they lost interest.” I Observation I

11. 11 Can workers guess which types of content are available? HITs with and without term clouds Several variations of term clouds E Projection Into Different Roles

12. 12 Jim and his large circle of friends have a huge collection of files that they are sharing with a very popular file-sharing program. The file-sharing program is a make-believe program. Please imagine that it looks something like this sketch: F Projection Into Different Roles

13. 13 (1)If you could download one of these files, which one would it be? (1)Why would you choose this particular file for download and viewing? (1)Think again about the file that you chose. Why did you guess that Jim or one of his friends would have this file in their collection? V Questions

14. 14 Some workers match literally between the mock-up frame and the questions The majority of workers is able to generalize from the situation or the mock-up frame Hypothesis: “HIT design can enhance the workers' success at completing projection tasks.” I Observation II

15. 15 Answer quality and HIT uptake of under the influence of Title Questionnaire design 5 experimental conditions: 5 HITs per condition 10 workers per HIT $0.10 reward per assignment E Further Investigation

16. 16 Title conditions 1)A: Jim, his friends and a make-believe file-sharing program 2)B: Jim, his friends and digital stuff to download 3)C: Jim, his friends and interesting stuff to download E Further Investigation

17. 17 All HITs yielded serious results (two assignments rejected due to cheating) Title A: more than 2 days to complete Title B+C: each completed within a day R Further Investigation

18. 18 Question conditions (using Title B): 1)No preference questions at all, only an explanation of all given judgments 2)No justification of preference asked E Further Investigation

19. 19 Absence of preference questions: Cut and paste strategies possible due to generality of question Serious answers become more verbose to capture the generalized situation No explanation of preference: Slight decrease in the degree to which the workers managed to project into Jim and his friends R Further Investigation

20. 20 “High imaginative load” tasks can be successfully run on MTurk The key appears to be a combination of: Signaling to workers the unique nature of the tasks (which are possibly quite different from those they generally choose) results in faster HIT uptake Making each HIT require individualized free-text justification response improves the workers ability of projection and generalization C Conclusions

21. 21 Q Questions?