1. Many hands make light work…
Or do they? Results of two pilot studies looking at the effects of crowdsourcing

2. No commercial conflicts of interest to declare

3. Crowdsourcing is…
…the practice of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees or suppliers.
…tasks are outsourced to an undefined public

4. Crowdsourcing in action
Amazon Mechanical Turk
The Sheep Market: sheep

5. Using the crowd for science
“Dive on in”
Zooniverse

6. Performing tiny ‘nano’ classification tasks on huge, on-going datasets
Biology
Perhaps one day we’ll see Cochrane Zoo!
Performing tiny ‘nano’ classification tasks on huge, on-going datasets
Sounds familiar…

7. In Cochrane Proposed nano task: first assess of search results?
One BIG task
Smaller tasks
Nano tasks
Clinically relevant question which needs an answer
Proposed nano task: first assess of search results?

8. 2 studies – 1 presentation
Trial Blazers
MoCAsub

9. Trial Blazers MoCAsub CENTRAL a DTA review
Background: Both studies looked at the efficiency of using a ‘crowd’ to screen search results for possible inclusion in:
CENTRAL
a DTA review

10. Trial Blazers
MoCAsub
In both studies we wanted to assess the following: Sensitivity: did they keep everything in that they should have? Specificity: did they reject all the records that should have been rejected?

11. Trial Blazers MoCAsub Recruited through: Recruited through:
Consecutive sample:
First 100 to sign up
No prior experience of EMB, SR, Dementia needed
Recruited through:
Consecutive sample: 20
No prior experience of EMB, SR, Dementia needed

12. Trial Blazers MoCAsub Baseline characteristics Mean age: 33
Age range: 16-63
Perceived knowledge:
38% no knowledge of EBM
38% a little knowledge of EBM
21% a lot of knowledge of EBM
Mean age: 27
Age range: 23-43
Perceived knowledge:
8% no knowledge of DTA
84% a little knowledge of DTA
8% a lot of knowledge of DTA

13. Methods: Trial Blazers
Designed and built by Kit Huckvale
Each participant screened the same 200 citations

14. Each participant screened the same 250 citations
Methods: MoCAsub
Each participant screened the same 250 citations

15. How to measure performance
Reference standard:
The reference standard was me, a TSC
Trials search co-ordinator: me
Reference standard:
The reference standard was the expert author team
Lead authors: clinical experts

16. Did it work?

17. Results: participant flow
100
Expressed an interest to take part 47
Didn’t/failed to register their device 53
Registered their device with TestFlight 28
Didn’t/failed to download ScreenToGo 25
Devices approved and downloaded ScreenToGo
High drop-out with a crowd model is often cited, but as long as you get enough – does it matter? And are you likely to get better results with those who don’t drop-out? 5
Did the demo but not the task 20
Did the task

18. Results: quantitative
Sensitivity: Specificity: 93.6

19. Results: quantitative
Sensitivity: Specificity: 60.9

20. Results: comparing the two
Excellent specificity
Misclassification of post-hoc analyses
30 records for CENTRAL
Excellent sensitivity
“if you’re not sure:
KEEP IT IN”
32 records for further assessment; 10 went on for inclusion

21. Results: qualitative Trial Blazers % MoCAsub % 30 70 8 92 100 61 31 23
How well did you think you did?
Identified all:
Identified most: 30 70 8 92
Would you like to know how well you did?
Yes:
No:
100
How difficult did you find the task?
Quite difficult:
Quite difficult; got easier:
Not difficult at all: 61 31 23 54
Was the guidance helpful?
Very helpful:
Quite helpful: 46
Did you feel you learnt anything?
Yes, a lot:
Yes, a bit: 77

22. Results: qualitative
Would you do this again?
But Why?

23. Results: motivations for taking part
“My wife made me do it”

24. Results: motivations for taking part
“My wife made me do it”
World Peace

25. Results: motivations for taking part
“My wife made me do it”
Having had first hand experience of having a relative with dementia, and the devastation this causes to family life, I am very keen to be involved in and to assist in, dementia research, even in a small way
World Peace

26. Results: motivations for taking part
To learn about research and for my cv!
To further my understanding of meta-analyses and systematic reviews, gain experience in Cochrane work
“a CV booster”
EXPERIENCE OF COCHRANE PROCESS. CV POINTS! CV
I would like more involvement in research - to improve my knowledge and skills in this area
To have participated in some form of research - as part of my core training in Psychiatry - which will be recognised on my CV at future job interviews
Wishing to help further dementia research in any way possible, thought it might help improve my knowledge of evidence based medicine, thought it might be helpful for my CV
I am interested in dementia and the work that the Cochrane Collaboration does and thus thought that this was a great opportunity

27. Summary: some challenges
Effectiveness and Quality – tricky balance between sensitivity and specificity
Guidance – how much of the broader picture vs “dive on in”
Feedback – how to do this with a ‘live’ task?

28. 3 key ingredients
Motivated
Supported
Slick
Versatile
Needed
Doable

29. Screen 5 citations per day x 31 000 contributors
Five a day
Screen 5 citations per day x contributors

30. Authors
Anna Noel-Storr Caroline Struthers Rupert McShane Sarah Cullum Daniel Davis Sam Creavin Gordon Dooley Kit Huckvale

31. And thank you to all those who took part
For more information please