1. Searching & screening efficiency: emerging techniques for systematic review Alison Weightman Co-convenor Cochrane Information Retrieval Methods Group

2. This presentation Searching for studies – Minimise the number of databases searched? – Maximise the value of search terms → text mining? Screening studies – Boost relevant records to the top of the list → text mining? – Use two screens Emphasis on text mining for search development …. and on free software

3. Acknowledgement: Selected text analysis tools Julie Glanville Associate Director, York Health Economics Consortium, University of York, UK

4. Stages of a systematic review Step 1: Develop a protocol Step 2: Search for the evidence Step 3: Screen and select studies Step 4: Assess the evidence Step 5: Extract the data Step 6: Synthesise the evidence Step 7: Present findings

5. Step 2: Search for the evidence – traditional steps Manual search of……. Multiple databases with lots of search terms many 000s hits Plus supplementary searching Web sites Reference lists Citation tracking Expert contact

6. Step 2: Search for the evidence – new approaches 2.1 Minimise the number of databases.... analyse previous reviews? 2.2 Maximise the value of search terms.... use text mining?

7. Morgan EBLIP 2015….. Searches for public health intervention reviews should include Medline/PubMed, reference list checking and citation tracking along with a small number of specially selected additional databases and expert contact/specialist web site searching 2.1 Minimise the number of databases?

8. Deriving high-quality (useful) information from unstructured text. The text words in relevant publications are grouped or structured to find word combinations that can be used to locate other similar materials 2.2 Maximise the value of search terms?..consider text mining

9. 2.2 Text mining to maximise the value of search terms O’Mara-Eves 2014: ‘We highly recommend that reviewers use automatic term extraction to identify additional search terms, particularly where the topic is broad or multidisciplinary’

10. 2.2 Text mining for search development: A worked example A systematic review of musculoskeletal care pathways for adults with hip and knee pain at the interface between primary and secondary care Three main concepts for search strategy: Musculoskeletal + pathway + primary- secondary interface

11.  Most common text words  PubMed PubReMiner, GoPubMed  Endnote  Most common phrases  Termine  Most common indexer-defined subject headings (MeSH)  MeSHonDemand  GoPubMed  Yale MeSH analyser Some text analysis tools

12. http://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi Example topic 1.Musculoskeletal primary care referral: Musculoskeletal + care pathway + primary- secondary 12 relevant papers. PubMed identifiers 23561693, 25442485, 25242531, 21056164, 25410703, 15576743, 10848384, 24337981, 25342909, 22716771, 23750266, 22711378 PubMed PubReMiner

13. Musculoskeletal, care (text) Musculoskeletal diseases, referral and consultation (MeSH)

14. Initial terms suggested by PubReMiner Musculo- skeletal Title/abstract words: Musculoskeletal Medical Subject Heading, MeSH: Musculoskeletal diseases/ Care Pathway Title/abstract words: Care Primary- Secondary care Medical Subject Heading, MeSH: Referral and consultation/

15. http://www.nactem.ac.uk/software/termine/ Example topic Musculoskeletal primary care referral: Musculoskeletal + care pathway + primary-secondary 12 relevant papers. Abstracts Enter all 12 abstracts in full BACKGROUND: Musculoskeletal condition assessment and management is increasingly delivered at the primary to secondary care interface, by inter- disciplinary triage and treat services. OBJECTIVES: This review aimed to describe Intermediate Care pathways, evaluate effectiveness, describe outcomes and identify gaps in the evidence…………… Termine combined with visual scan of abstracts

16. Terms scoring ≥3 RankTermScore 1conversion rate8.75 2primary care7 3secondary care5 3musculoskeletal service5 5patient satisfaction4.333333 6systematic review4 6musculoskeletal disorder4 8health care service3.169925 8surgery conversion rate3.169925 8uncomplicated musculoskeletal problem3.169925 11scope physiotherapist3 11treatment effectiveness3 11orthopaedic consultation3 11orthopaedic surgeon3

17. Search v2 after Termine analysis and visual scan of abstracts Musculo- skeletal Title/abstract words: Musculoskeletal, orthop?edic Medical Subject Heading, MeSH: Musculoskeletal diseases/ Care pathway Title/abstract words: Care adj2 (pathway* or plan* or protocol* or map*) Primary to secondary Title/abstract words: Conversion rate, primary adj2 secondary, scope physiotherap*, orthop?edic adj (consulta* or surgeon), Refer adj3 (GP or general practi* or primary care) Medical Subject Heading, MeSH: Referral and consultation/

18. http://mesh.med.yale.edu/ Example topic Musculoskeletal primary care referral: Musculoskeletal + care pathway + primary- secondary 12 relevant papers. PubMed identifiers 23561693, 25442485, 25242531, 21056164, 25410703, 15576743, 10848384, 24337981, 25342909, 22716771, 23750266, 22711378 Yale MeSH analyser

19. Musculoskeletal pain (MeSH) Critical pathways (MeSH)

20. Search v3 after adding in Medical Subject Headings via Yale MeSH analyser Musculo- skeletal Title/abstract words: Musculoskeletal, orthop?edic Medical Subject Headings, MeSH: Musculoskeletal diseases/, Musculoskeletal pain/ Care pathway Title/abstract words: Care adj2 (pathway* or plan* or protocol* or map*) Medical Subject Headings, MeSH: Critical pathways/ Primary to secondary Title/abstract words: Conversion rate, primary adj2 secondary, scope physiotherapy*, orthop?edic adj (consulta* or surgeon), Refer adj3 (GP or general practi* or primary care) Medical Subject Headings, MeSH: Referral and consultation/, Triage/, Ambulatory care/, Ambulatory care facilities/

21. Final draft strategy v4 incorporating expert view, scoping searches and tests for sensitivity/precision balance MusculoskeletalTitle/abstract words: Musculoskeletal, orthop?edic, hip, knee Medical Subject Headings, MeSH: Musculoskeletal diseases/, Musculoskeletal pain/ Care pathway Title/abstract words: Care adj2 (pathway* or plan* or protocol* or map*), Clinic* adj3 (standard* or pathway* or assessment* or map* or protocol*), (Critical or patient*) adj3 pathway*, (triage adj3 (service* or process* or system*)), (screening adj3 (service* or standard* or protocol*)) Medical Subject Headings, MeSH: Critical pathways/, Triage/ Primary to secondary Title/abstract words: Conversion rate, primary adj2 secondary, extended scope adj3 physiotherapy*, orthop?edic adj (consulta* or surgeon), Refer adj6 (GP or general practi* or primary care or outpatient or consultation), assessment adj clinic*, triage, MCAT, MCASS, MCAS, CATS Medical Subject Headings, MeSH: Referral and consultation/, Triage/, Ambulatory care/, Ambulatory care facilities/ AND OR

22. Search efficiency 256 hits in Ovid Medline Circa 1/5 hits potentially relevant (select for full text) Finds >90 % relevant papers (development set) Finds >70% papers (test set) 50% of test set were not in Medline Additional databases (including physio) and extensive supplementary searching required Then further test for sensitivity

23. Stages of a systematic review Step 1: Develop a protocol Step 2: Search for the evidence Step 3: Screen and select studies Step 4: Assess the evidence Step 5: Extract the data Step 6: Synthesise the evidence Step 7: Present findings

24. Step 3: Screen studies – traditional steps Manual human process involving… Knowledge of inclusion criteria – Identified in protocol Selection – Title and abstract – Full text Screen and select independently in duplicate – Reduction of error and bias

25. Step 3: Screen and select studies – new approaches 3.1 Boost relevant records to the top of the list…. Text mining? 3.2 Reduce screening time ….. Two screens? http://www.cs.manchester.ac.uk/our-research/research-groups/text-mining /

26. Weightman 2014…if the reviewer is willing to accept that some titles/abstracts may not be picked up but that all full text includes will be found, benefits may be huge and could reduce the screening burden by up to 70%. An enormous efficiency saving where a large number of results need to be screened. 3.1 Faster screening: EPPIReviewer …Free soon to Cochrane review authors

27. Rathbone 2015….Abstrackr has the potential to save time and reduce research waste. [Of four reviews one relevant citation was missed in each of two reviews] 3.1 Faster screening: Abstrackr http://abstrackr.cebm.brown.edu/

28. Wang 2014…..Using dual monitors when conducting SRs is associated with a significant reduction of time spent on data extraction (~24 minutes per article). No significant difference was observed in time spent on abstract screening or full text screening 3.2 Reduce screening time ….. Two screens? http://northerncomputer.blogspot.co.uk/2011/11/many-people-spend-good-part-of-their.html

29. Summary: Choice of databases: Carefully chosen databases – and emphasise the additional searching techniques (reference lists, citation tracking, expert contact…) Text mining for searching and screening: Wide choice of free software Two screens: No need for screening but yes for data extraction….. I use for both (ed!)

30. Any questions or suggestions WeightmanAL@cardiff.ac.uk

31. References Morgan HE et al. Systematic reviews of public health interventions to support practice & policy: Where should you look? [Poster] Eight International Evidence Based Library and Information Practice Conference (EBLIP8), Brisbane Australia, July 2015 O’Mara-Eves A et al. Techniques for identifying cross-disciplinary and ‘hard-to- detect’ evidence for systematic review. Research Synthesis Methods 2014; 5(1): 50-59Techniques for identifying cross-disciplinary and ‘hard-to- detect’ evidence for systematic review Weightman A et al. Text mining for screening efficiency? Testing within a Cochrane public health review. [Poster] 23 rd Cochrane Colloquium, Hyderabad India, September 2014Text mining for screening efficiency? Testing within a Cochrane public health review Rathbone J et al. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews 2015; 4: 80Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers Wang Z et al. Dual computer monitors to increase efficiency of conducting systematic reviews. Journal of Clinical Epidemiology 2014; 67(12): 1353-1357Dual computer monitors to increase efficiency of conducting systematic reviews