1. Issues and Challenges around Searching the Literature Dr Andrew Booth

2. Overview What is Already Known on Searching the Qualitative Research Literature Overview of Methodological Issues/Challenges Recent Developments (with focus on literature of last two years) Outstanding Issues/Challenges

3. What is already known

4. What is Already Known? – 1 Finding relevant qualitative studies arduous due to inadequate indexing (Dixon Woods et al, 2007; Ring et al 2011) Papers often lack abstracts or include non-informative titles, making it difficult to establish relevance (Dixon Woods et al, 2007). Several papers outline strategies (filters) for searching for qualitative studies ( Walters et al, 2006; Wilczynski et al, 2007; Wong et al, 2004; McKibbon et al, 2006 ) Helpful techniques involve electronic or hand searching (Greenhalgh et al 2005; Bates, 1989): – Reference or footnote tracking (looking backwards at references in articles found). – Citation tracking (tracking forward subsequently citing articles). – Personal knowledge and personal contacts. – Contacting the authors of known papers or experts in the field. – Hand searching relevant journals. – Internet browsing such as berry picking (where one search leads to another and ‘clusters’ of papers are often found together). Greenhalgh et al (2005) found only 30% of primary sources from predefined search strategy. 51% found by other predefined methods (i.e. reference, footnote and citation tracking). However this may reflect a priori decisions to concentrate on non-electronic sources

5. What is Already Known? - 2 “Literature searches…open-ended iterative processes where the topic or research question of interest is honed over time as the nature of the evidence becomes more apparent” (Finfgeld-Connett & Johnson 2013) Number of articles not only critical factor – reports “may lack enough thick description to fully develop concepts and the interrelationships among them” (Finfgeld-Connett & Johnson 2013) Unpublished studies may contain rich, thick description Goal may not be aggregative – theoretical saturation may play a part – selection of sample is crucial

6. References for What is Known Already - 1 Bates MJ: The design of browsing and berrypicking techniques for on-line search interface. Online Rev 1989, 13:407-424. Dixon-Woods M, Bonas S, Booth A et al.: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res 2006, 6:27-44. Finfgeld‐Connett, D., & Johnson, E. D. (2013). Literature search strategies for conducting knowledge‐building and theory‐generating qualitative systematic reviews. Journal of Advanced Nursing, 69(1), 194-204. Greenhalgh T, Peacock R: Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005, 331:1064-1065. Ring N, Jepson R, Ritchie K: Methods of synthesising qualitative research for health technology assessment. Int J Technol Assess Health Care 2011, 27:384-390

7. References for What is Known Already - 2 Walters LA, Wilczynski NL, Haynes RB: Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res 2006, 16:162- 168. PubMed Abstract | Publisher Full Text PubMed AbstractPublisher Full Text Wilczynski NL, Marks S, Haynes RB: Search strategies for identifying qualitative studies in CINAHL. Qual Health Res 2007, 17:705- 710. PubMed Abstract | Publisher Full Text PubMed AbstractPublisher Full Text Wong SL, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. In Medinfo 2004: Proceedings of the 11th World Congress on Medical Informatics; San Francisco. Edited by Fieschi M, Coiera E, Jack Li YC. Amsterdam: IOS Press; 2004:311-314. McKibbon KA, Wilczynski NL, Haynes RB: Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Eval Health Prof 2006, 29:440-454. PubMed Abstract | Publisher Full Text PubMed AbstractPublisher Full Text

8. Overview of Methodological Issues/Challenges Bias towards/Predominance of quantitative research and publication of resultant reports Non-optimal indexing of qualitative studies (CINAHL more evolved than MEDLINE) Qualitative research represents various research methodologies, including ethnography, phenomenology, grounded theory and narrative analysis, which may hinder retrieval Lack of informative manuscript titles and abstracts

9. Overcoming barriers: Searches should be as transparent as possible without jeopardizing the creativity and complexity of the process (Finfgeld-Connett & Johnson 2013) Key “test” for knowledge-building/theory generating review – would small amounts of conflicting information substantially change the findings? [Theoretical Saturation/ Qualitative Sensitivity Analysis] Weaknesses in indexing mean that sensitivity of searches may need to be reduced to allow time for other search strategies (Pearson et al, 2011)

10. Simple search strategies vs complex ones Three broad-based terms (i.e. “qualitative”, “findings” and “interviews”) as effective as more complex search strategies in identifying relevant qualitative research reports (Flemming & Briggs 2007). Search strategies with broad search terms (“qualitative research” or “qualitative studies” or “interviews”) [combined with CAM terms] had highest recall and precision (Franzel, 2013). Within time-limited context, protocol-driven, targeted, and reference-checking search strategies most effective (Pearson et al, 2011)

11. Intervention Searching vs Condition Searching (Lorenc et al, 2012) Tying search terms of SR of qualitative evidence too closely to interventions may compromise consistency of the review. Dilemma: Performing condition-wide searches (with no other change to strategies) would become highly over-inclusive and volumes of records impracticably large. (Suggests need for alternative sampling stategies)

12. Is More Necessarily Better? For knowledge-building and theory-generating systematic reviews, “more is better only when it helps to fully explicate a concept, substantiate an interconnection between or among concepts, or build a line of argument….collecting more of the same may merely escalate the cost of a study, clutter the database and obfuscate important inferences. Concepts and interrelationships among them can only be more fully explicated based on data that adds depth, breadth, meaning and understanding to a phenomenon” (Finfgeld-Connett & Johnson 2013).

13. Searching for religion and mental health studies required health, social science, and grey literature databases (Wright et al, 2014) PsycINFO = best performing database ArabPsyNet, CINAHL, Dissertations and Theses, EMBASE, Global Health, Health Management Information Consortium, MEDLINE, PsycINFO, and Sociological Abstracts essential to retrieve included references. Citation tracking and personal library of one of the research teams contributed unique, relevant references. Religion studies databases (Am Theo Lib Assoc, FRANCIS) did not provide unique, relevant references. Literature searches for reviews/evidence syntheses of religion and health studies should include social science, grey literature, non-Western databases, personal libraries, and citation tracking activities.

14. Exhaustive versus Expansive That is the Question!

15. Exhaustive versus Expansive (Finfgeld- Connett & Johnson 2013) Exhaustive searches when conducting summative and aggregative systematic reviews Expansive searches when conducting knowledge-building and theory-generating systematic reviews

16. Exhaustive versus Expansive Multiple databases Comprehensive list of terms Assumption of homogeneity Other approaches are “supplementary” Databases to reflect contributing disciplines Terminology may characterise “narrative” of each discipline Likelihood of divergent cases Sibling studies (shared context) Cluster searching Citation searching “Supplementary” techniques may be more important “Combing the Area” “Following Up Leads”

17. Recent Developments Importance of Context/Theory Database Coverage Importance of Supplementary Search Methods Appropriate Selection of Sampling Methods Reporting Standards (e.g. ENTREQ)

18. Searching for Contextual Richness Requires identification of related (sibling) reports i.e. cluster searching (cluster becomes unit of analysis, not study) (Booth et al, 2013)

19. Cluster searching for “Siblings” (Booth et al, 2013) ElementProcedural Steps Citations Backwards reference chaining Lead Authors Author searching; Backwards reference chaining Unpublished materials Web searches; repositories Scholar searches Citations Theories Backwards reference chaining Early Examples Cited works (Forward reference chaining) Related Projects Co-citations

20. Searching for Theories Theory not typically reported in Abstracts Theoretical base differs by discipline cp. HSR vs Public Health vs Nursing vs Psychology vs Sociology Reporting of Theory differs by discipline Level of Theory may vary e.g. Individual versus Society (Psychology vs Sociology) When is a “theory” a Theory? – labelling (model, framework, concepts ) and naming (“Health Belief Model”)

21. Database Coverage CAM search - PubMed yielded 87% of relevant included qualitative studies (Franzel et al, 2013). Five different QES PubMed coverage values 35/44 (79.5%); 9/10 (90%); 10/11 (91%); 9/9 (100%); 7/28 (25% - Grey literature) (Booth 2012 [Unpublished]) But 5/28 of studies located only through supplementary searches of three sources (Stansfield et al, 2012). 21 search sources required to locate all studies. [Explanation - Role of Grey Literature and Geographical focus – UK only]

22. Where to find qualitative research? 61% of [qualitative] research studies published within journals. 29% were published as research reports and 5% were published in books. Remainder were theses, conference papers and raw datasets. 66% of studies (66%) located across 19 bibliographic databases. 15 databases provided studies that were not identified elsewhere. PubMed a good source for all reviews. Supplementary information sources important for identifying studies in all publication formats. Undertaking sensitive searches across range of information sources is essential for locating views studies in all publication formats. (Stansfield et al, 2014)

23. Supplementary Strategies More diffuse topic, move beyond electronic searching

24. Supplementary Strategies - 1 Multiple search strategy more likely to identify relevant QR than sole reliance on electronic searching. Purpose of synthesis determines appropriate sampling/search strategy. E.g. mapping out key conceptual developments – if aim not aggregative, omission of papers unlikely to have dramatic effect on results. Difficult to maintain sufficient familiarity with > 40 papers (Campbell et al, 2011)

25. Supplementary Strategies - 2 Need ‘belt and braces’ (hand-searching; consultation with experts) Searching for books/theses particularly challenging (not indexed in same way as journal papers) (Campbell et al, 2011) Snowballing and consultation with experts for a realist review (Pawson et al, 2004) Obtaining authors' suggestions - resource-intensive process with negligible results (Pearson et al, 2011) Additional search techniques essential to locate further high quality references (Papaioannou et al, 2010)

26. Sampling: Appropriate ≠ Comprehensive (Suri, 2011) 16 strategies for sampling in QES E.g. Snowball sampling - seeking information from key informants about other ‘information-rich cases’. – ‘Chain of recommended informants would typically diverge initially as many possible sources are recommended, then converge as a few key names get mentioned over and over’ (Patton, 2002, p. 237). Identify most cited primary research reports by ‘footnote chasing’ (searching citation indices, browsing through bibliographies, previous research syntheses, primary research reports, policy documents, papers written by practitioners and papers written for practitioners).

27. ENTREQ & Searching (Tong et al, 2012) 3 Approach to searching Indicate whether search was pre-planned (comprehensive search strategies to seek all available studies) or iterative (to seek all available concepts until they theoretical saturation is achieved). 4Inclusion criteria Specify inclusion/exclusion criteria (e.g. in terms of population, language, year limits, type of publication, study type). 5Data sources Describe information sources used (e.g.electronic databases (MEDLINE, EMBASE, CINAHL, psycINFO, Econlit), grey literature databases (digital thesis, policy reports), relevant organisational websites, experts, information specialists, generic web searches (Google Scholar) hand searching, reference lists) and when searches conducted; provide rationale for using data sources. 6 Electronic Search strategy Describe literature search (e.g. provide electronic search strategies with population terms, clinical or health topic terms, experiential or social phenomena related terms, filters for qualitative research, and search limits).

28. Improvement in Search Reporting 1988-20042005-2008 (Hannes & Macaitis, 2012)

29. Outstanding Issues/Challenges How to understand publication bias in qualitative research (not around positive/negative findings)? How to systematise (and document) more intuitive search approaches e.g. cluster searching and searching for theory? How does sampling strategy translate into search strategy? How to construct sampling frames for studies? How to sample for diversity? How many sources are enough? How to retrieve rich data? How to retrieve data on theory and context?

30. Publication Bias in Qualitative Research? – Part One “This does not mean that publication biases do not exist in….qualitative research… …qualitative studies may be less frequently conducted, submitted for publication and/or published in high quality and easily accessible journals….raw data (i.e. research findings)….needed to conduct a qualitative systematic review may not be readily available” (Finfgeld-Connett & Johnson 2013)

31. Publication Bias in Qualitative Research? – Part Two “ignoring grey literature, such as dissertations/theses, government reports, monographs and books, on the basis that it may be of lesser quality (as with quantitative research) is empirically and logically invalid…..[such] documents may be particularly rich sources of qualitative data as page limits are not generally imposed...”. (Finfgeld-Connett & Johnson 2013) Qualitative researchers often choose to publish in book form (“Truncation bias”)

32. Conclusions Increasing Importance of Explanatory Sources (e.g. Context and Theory) Need for Ongoing Investigation of Database Coverage and Supplementary Search Techniques Requires Exploration/Selection of Appropriate Sampling Methods Value versus Effort Trade-Off

33. References - 1 Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015 Jun 11. doi: 10.1111/hir.12108. Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013 Sep 28;13:118. Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, et al. Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess 2011; 15(43). Flemming, K., & Briggs, M. (2007). Electronic searching to locate qualitative research: evaluation of three strategies. Journal of Advanced Nursing, 57(1), 95-100.

34. References - 2 Franzel, B., Schwiegershausen, M., Heusser, P., & Berger, B. (2013). How to locate and appraise qualitative research in complementary and alternative medicine. BMC complementary and alternative medicine, 13(1), 125. Guise, J. M., Chang, C., Viswanathan, M., Glick, S., Treadwell, J., Umscheid, C. A.,... & Trikalinos, T. (2014). Systematic Reviews of Complex Multicomponent Health Care Interventions [Internet]. http://www.ncbi.nlm.nih.gov/books/NBK194851/ http://www.ncbi.nlm.nih.gov/books/NBK194851/ Hannes, K., & Macaitis, K. (2012). A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qualitative Research, 12(4), 402-442. McGinn, T., Taylor, B., McColgan, M., & McQuilkan, J. (2014). Social Work Literature Searching Current Issues With Databases and Online Search Engines. Research on Social Work Practice, 1049731514549423. Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information & Libraries Journal, 27(2), 114-122. Pearson, M., Moxham, T., & Ashton, K. (2011). Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Evaluation & the Health Professions, 34(3), 297-308.

35. References - 3 Stansfield, C., Brunton, G., & Rees, R. (2014). Search wide, dig deep: literature searching for qualitative research. An analysis of the publication formats and information sources used for four systematic reviews in public health. Research Synthesis Methods, 5(2), 142-151. Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The selection of search sources influences the findings of a systematic review of people’s views: a case study in public health. BMC Medical Research Methodology, 12(1), 55. Suri, H. (2011). Purposeful sampling in qualitative research synthesis. Qualitative Research Journal, 11(2), 63-75. Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology, 12(1), 181. Suri, H. (2013). Towards Methodologically Inclusive Research Syntheses: Expanding Possibilities. Routledge. Wright JM, Cottrell DJ, Mir G. Searching for religion and mental health studies required health, social science, and grey literature databases. J Clin Epidemiol. 2014 Jul;67(7):800-10.