Many hands make light work… Or do they? Results of two pilot studies looking at the effects of crowdsourcing

No commercial conflicts of interest to declare

Crowdsourcing is… …the practice of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees or suppliers. …tasks are outsourced to an undefined public

Crowdsourcing in action Amazon Mechanical Turk The Sheep Market: 10 000 sheep

Using the crowd for science “Dive on in” Zooniverse www.zooniverse.org

Performing tiny ‘nano’ classification tasks on huge, on-going datasets Biology Perhaps one day we’ll see Cochrane Zoo! Performing tiny ‘nano’ classification tasks on huge, on-going datasets Sounds familiar…

In Cochrane Proposed nano task: first assess of search results? One BIG task Smaller tasks Nano tasks Clinically relevant question which needs an answer Proposed nano task: first assess of search results?

2 studies – 1 presentation Trial Blazers MoCAsub

Trial Blazers MoCAsub CENTRAL a DTA review Background: Both studies looked at the efficiency of using a ‘crowd’ to screen search results for possible inclusion in: CENTRAL a DTA review

Trial Blazers MoCAsub In both studies we wanted to assess the following: Sensitivity: did they keep everything in that they should have? Specificity: did they reject all the records that should have been rejected?

Trial Blazers MoCAsub Recruited through: Recruited through: Consecutive sample: First 100 to sign up No prior experience of EMB, SR, Dementia needed Recruited through: Consecutive sample: 20 No prior experience of EMB, SR, Dementia needed

Trial Blazers MoCAsub Baseline characteristics Mean age: 33 Age range: 16-63 Perceived knowledge: 38% no knowledge of EBM 38% a little knowledge of EBM 21% a lot of knowledge of EBM Mean age: 27 Age range: 23-43 Perceived knowledge: 8% no knowledge of DTA 84% a little knowledge of DTA 8% a lot of knowledge of DTA

Methods: Trial Blazers Designed and built by Kit Huckvale Each participant screened the same 200 citations

Each participant screened the same 250 citations Methods: MoCAsub Each participant screened the same 250 citations

How to measure performance Reference standard: The reference standard was me, a TSC Trials search co-ordinator: me Reference standard: The reference standard was the expert author team Lead authors: clinical experts

Did it work?

Results: participant flow 100 Expressed an interest to take part 47 Didn’t/failed to register their device 53 Registered their device with TestFlight 28 Didn’t/failed to download ScreenToGo 25 Devices approved and downloaded ScreenToGo High drop-out with a crowd model is often cited, but as long as you get enough – does it matter? And are you likely to get better results with those who don’t drop-out? 5 Did the demo but not the task 20 Did the task

Results: quantitative Sensitivity: 81.2 Specificity: 93.6

Results: quantitative Sensitivity: 100.0 Specificity: 60.9

Results: comparing the two Excellent specificity Misclassification of post-hoc analyses 30 records for CENTRAL Excellent sensitivity “if you’re not sure: KEEP IT IN” 32 records for further assessment; 10 went on for inclusion

Results: qualitative Trial Blazers % MoCAsub % 30 70 8 92 100 61 31 23 How well did you think you did? Identified all: Identified most: 30 70 8 92 Would you like to know how well you did? Yes: No: 100 How difficult did you find the task? Quite difficult: Quite difficult; got easier: Not difficult at all: 61 31 23 54 Was the guidance helpful? Very helpful: Quite helpful: 46 Did you feel you learnt anything? Yes, a lot: Yes, a bit: 77

Results: qualitative Would you do this again? But Why?

Results: motivations for taking part “My wife made me do it”

Results: motivations for taking part “My wife made me do it” World Peace

Results: motivations for taking part “My wife made me do it” Having had first hand experience of having a relative with dementia, and the devastation this causes to family life, I am very keen to be involved in and to assist in, dementia research, even in a small way World Peace

Results: motivations for taking part To learn about research and for my cv! To further my understanding of meta-analyses and systematic reviews, gain experience in Cochrane work “a CV booster” EXPERIENCE OF COCHRANE PROCESS. CV POINTS! CV I would like more involvement in research - to improve my knowledge and skills in this area To have participated in some form of research - as part of my core training in Psychiatry - which will be recognised on my CV at future job interviews Wishing to help further dementia research in any way possible, thought it might help improve my knowledge of evidence based medicine, thought it might be helpful for my CV I am interested in dementia and the work that the Cochrane Collaboration does and thus thought that this was a great opportunity

Summary: some challenges Effectiveness and Quality – tricky balance between sensitivity and specificity Guidance – how much of the broader picture vs “dive on in” Feedback – how to do this with a ‘live’ task?

3 key ingredients Motivated Supported Slick Versatile Needed Doable

Screen 5 citations per day x 31 000 contributors Five a day Screen 5 citations per day x 31 000 contributors

Authors Anna Noel-Storr Caroline Struthers Rupert McShane Sarah Cullum Daniel Davis Sam Creavin Gordon Dooley Kit Huckvale

And thank you to all those who took part For more information please email: anna.noel-storr@rdm.ox.ac.uk