1. Qualitative Evidence Synthesis (QES): Karin Hannes Centre for Methodology of Educational Research

2. Qualitative Evidence Synthesis: Agenda PART 1 Its context (and how I became triggered by QES) How to formulate questions and search for qualitative insights PART 2 Its popularity Its role in systematic reviews (practical examples) A list of developed approaches A comparison of characteristics of two commonly used approaches PART 3 New developments – Context specific versus multi-context reviews – Mixed methods reviews

3. CONTEXT AND STEPWISE APPROACH PART 1:

4. CONTEXT Emma, Born the 6th of October 2010

5. CONTEXT

6. Women who exercised did not lose significantly more weight than women in the usual care group. Women who took part in a diet or diet plus exercise programme, lost significantly more weight than women in the usual care. There was no difference in the magnitude of weight loss between diet and diet plus exercise group. The interventions seemed not to affect breastfeeding performance adversely. CONTEXT

7. A recent study in the Journal of the America College of Nutritian found that those who ate cereals where lower in weight compared to those who ate meat and eggs or skipped breakfast. CONTEXT

8. Logical, rational reasoning: IF a diet helps to loose weight after pregancy. IF cereals have proven to work well as a diet. THEN the consumption of cereals will lead to weight loss after pregnancy! After having consumed cereals for several months…………….. If it has proven itself to work for others… THEN WHY DOESN’T IT WORK FOR ME! CONTEXT

9. What is evidence? Evidence of ‘effectiveness’: the extent to which an intervention, when used appropriately, achieves the intended effect. Evidence of ‘feasibility’: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context. Evidence of ‘appropriateness’ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given. Evidence of ‘meaningfulness’: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population. CONTEXT

10. Systematic Reviews Well formulated question (PICO) Well defined search strategy (sensitive, all-inclusive) Objective selection proces (inclusion/exclusion criteria) Critical appraisal of methodological quality (risk of bias) Objective data-extraction (standard sheets) Synthesis of data Conclusion and recommendations consistent with data Meta-analysis: pooling results from individual studies Quant: RCT, … Therapeutic, diagnostic, economic, preventive,… CONTEXT

11. Systematic Reviews IF I am not interested in evidence of effectiveness, BUT in feasibility, appropriateness, meaningfulness,… Qualitative Evidence Synthesis: A process of summarizing qualitative research findings, either aggregative or interpretive, by comparing and analysing texts derived from multiple accounts of an event or situation as reported in basic qualitative research studies. Explore questions such as how do people experience illness? why does an intervention work (or not), for whom and in what circumstances…? what are the barriers and facilitators to accessing health care? what impact do specific barriers and facilitators have on people, their experiences and behavior? CONTEXT

12. Systematic Reviews Could I still use the methodology outlined for SR to answer these questions? Could I use it in the same way? 1.Question formulation  PICO becomes SPI(C)E 2.Searching  Sensitive (all-inclusive) versus Specific (Selective) 3.Critical appraisal  Proponents versus Opponents 4.Synthesis  Variety of Approaches 5.Recommendations  Not always the goal, could be building theory as well CONTEXT

13. QES: Stepwise approach PICO becomes SPICE IIntervention/(topic of) Interest: diet (C)Comparison:(training, placebo) EEvaluation:Attitude, view, opinion on... Elements that hinder the diet Impact of the diet on the general welbeing of the mother Opinions on how to best integrate the diet in daily family life PPerspective:Mothers in a post-natal situation SSetting: Western SPI(C)E ! For some qualitative questions there isn’t an intervention to be evaluated.

14. QES: Stepwise approach Searching qualitative evidence: problems Little result from searching the major databases – 30% database & handsearch – 50% ‘snowballing’ – 24% personal contacts (Greenhalgh, 2005) Major problems: – Bad indexing – Less developed and tested methodological filters – MeSH-term: qualitative research General rules: – Use methodological filters – Examine references – Use the related article features in major databases – Search for citations (backword and forward)

15. QES: Stepwise approach Searching qualitative evidence: keywords Technique: focus group, interview, action research, content analysis, thematic analysis... Software: Nudist OR Nvivo OR Atlas-ti Citations: Glaser and Strauss, Miles and Huberman,... Phenomenon: perceptions, attitudes, user views, standpoints, viewpoints, experiences,... Methodology: ethnographic, phenomenologic,… Type of data: stories, narratives, descriptions, themes, exploration, findings ( results)

16. QES: Stepwise approach Searching qualitative evidence: INTERTASC www.york.ac.uk/inst/crd/intertasc/index.htm www.york.ac.uk/inst/crd/intertasc/index.htm nursing methodology research OR (qualitative OR ethnolog* OR ethnog* OR ethnomethodolog* OR emic OR etic OR phenomenolog*[Title/Abstract]) OR (hermeneutic* OR heidegger* OR husserl* OR colaizzi* OR giorgi* OR glaser OR strauss [title/abstract]) OR (kaam* OR manen OR participant observ* OR constant compar* [title/abstract]) OR (focus group* OR grounded theory OR "narrative analysis" OR lived experience* OR life experience* [title/abstract]) OR (theoretical sampl* OR purposive sampl* OR ricoeur OR spiegelberg* OR merleau [title/abstract]) OR (metasynthes* OR meta-synthes* OR metasummar* OR meta-summar* OR metastud* OR meta-stud* [title/abstract]) OR (maximum variation OR snowball [title/abstract]) OR (field stud* OR field note* OR fieldnote* OR field record* OR action research [title/abstract]) OR (thematic analys* OR content analy* OR unstructured categor* or structured categor* [title/abstract]) OR (participant observation* OR nonparticipant observation* OR non participant observation* [title/abstract]) OR (tape recording OR "tape record*" OR "video record*" OR "audio record*" OR taperecord* OR audiorecord* OR videotap* OR videorecord*) http://www.library.ualberta.ca/subject/healthsciences/QualitativeFilters/index.cfm Methodology Citations/authors Techniques

17. QES: Stepwise approach Searching qualitative evidence: debate We need a sensitive approach to searching that includes all potentially relevant studies. We need a search strategy that is specific and purposeful, including studies that are relevant to our synthesis. This might include working with a saturation point for inclusion.  Related to the goal and/or approach of a particular study. Searching: addressed in upcoming presentation Next step ‘critical appraisal’ addressed in upcoming presentation

18. PART 2: Qualitative Evidence Synthesis (QES) approaches QES in the public health literature The role of QES (in SR) A list of developed approaches Comparing characteristics of different approaches

19. QES in the public health literature Timetrend based on the reviews from: Dixon-Woods & colleagues (2007) Hannes & Macaitis (2011)

20. Quantitative Accumulating Strictly comparable More power Through data Nature Aim Studies Result Synthesis Meta- analysis Evidence-synthesis Qualitative Make sense of data Basic comparability Added value in content Through interpretation The role of QES in SR: different aims

21. The role of QES in SR Inform: reviews by using evidence from qualitative research to help define and refine the question, and to ensure the review includes appropriate studies and addresses important outcomes (scoping review) Enhance: reviews by synthesizing evidence from qualitative research identified whilst looking for evidence of effectiveness (process and implementation issues). Extend: reviews by undertaking a search to specifically seek out evidence from qualitative studies to address questions directly related to the effectiveness review (mixed method or multilevel synthesis). Supplement: reviews by synthesizing qualitative evidence within a stand-alone, but complementary review to address questions on other than effectiveness (stand-alone or parallel synthesis). Note:These should be distinguished from a narrative report of a quantitative SR. When individual studies cannot be pooled quantitatively (reason: heterogeneity), they may still have useful qualitative information to be shared with the reader.

22. Narrative synthesis because pooling is not possible? Rather not labelled as QES… Example:Narrative report Adding a bit won’t work. The need: full SR of QR !

23. ‘Mixed Method Approach’ Children & Healthy Eating - EPPI-centre: eppi.ioe.ac.uk Children’s ViewsTrials Recommendation for interventionsGood qualityOther Do not promote fruit and vegetables in the same way None Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one 55 Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 56 Example: Extending Review Which interventions match recommendations derived from children’s views and experiences? Conclusion

24. Example: Supplementing review Barroso J, Powell-Cope GM. Metasynthesis of Qualitative Research on Living with HIV Infection. Qual Health Res vol 10, nr 3, 2000.: – Understand experience of adults living with HIV infection – 21 articles – Method: constant comparative analysis – Themes: Finding meaning in HIV: death, loss, surviving Shattered meaning: fear, irreparable Human connectedness: meaningfull relationships alienation, isolation Focusing on the self: actions to enhance fysical and emotional health Negociating care: active role Dealing with stigma: personal, family, society

25. The role of QES in SR Provide evidence on the subjective experience of those involved in developing, delivering and receiving an intervention OR/AND Provide a research-based context for interpreting and explaining trial results How to achieve change (more effectively)? How to improve interventions? How to ‘fit’ subjective needs? What other type of interventions might be needed? OR/AND Reveal the extent to which effective interventions are actually adopted in policies and practice (what are barriers and bridges?) OR/AND Contribute to the understanding of heterogeneity in outcomes

26. A list of developed approaches QES-approachDeveloped Meta-ethnographyNoblit & Hare, 1988 Meta-summarySandelowski & Barosso (2008) Meta-studyPaterson et al (2001) Realist synthesisPawson et al (2004) Meta-narrative mappingGreenhalgh (2005) Critical Interpretive SynthesisDixon-Woods et al (2006) Narrative SynthesisPopey et al (2006) Textual narrative synthesisLucas (2007) Ecological triangulationBanning (unknown) Framework synthesisBrunton et al (2006), Oliver et al (2008) Meta-interpretationWeed (2005) Meta-aggregationJoanna Briggs Institute (2001), Hannes and Lockwood (2010) Bayesian meta-analysisRoberts et al (2002), Voils et al (2009) Content analysisEvans and Fitzgerald (2002), Suikkala & Leino-Kilpi (2000) Case SurveyYin & Heald (1975), Jensen & Rodgers (2001) Qualitative Comparative analysisCress & Snow (2000) Thematic synthesisThomas & Harden (2008) Cross-case analysisMiles & Huberman (1994) Grounded theoryFinfgeld (1999) Kearney (2001), Eaves (2001) ……

27. A list of developed approaches MAKING SENSE OF THE MYRIAD OF QUALITATIVE EVIDENCE SYNTHESIS METHODS No clear guidance about how to determine which of the existing synthesis methods best fits a particular purpose. possible considerations…

28. A list of developed approaches ConsiderQuestions to be asked The nature of the research Will the synthesis method result in the expected and desired outcomes? Is the method congruent with the goals of the synthesis project? Does the primary research support the method? The nature of the researcher How tolerant is the researcher to the amount of structure and ambiguity that is inherent in the method? Is his/her epistemological stance congruent with that of the synthesis method? The nature of the research team Is there the necessary mix of disciplinary, methodological, and other perspectives among the research team to enact this method? Is the expertise needed for this method (e.g., statistical analysis, theoretical) available? Resource requirements How much personnel, time and effort are required for this method? Is there adequate funding to support expenses incurred in implementing this method? Paterson, B. (2011). Introducing Qualitative Evidence Synthesis. In Hannes, K., & Lockwood, C. (eds.). Qualitative Ev. Synthesis: choosing the right approach. Wiley- Blackwell, UK.

29. Qualitative Evidence Synthesis approaches Decision to conduct a qualitative evidence synthesis To aggregate / summarise / integrate qualitative data to address specific questions in relation to a Cochrane intervention review To interpret synthesised qualitative evidence and develop explanatory theory or models Primarily to integrate and interpret qualitative and quantitative evidence within a single approach or integrated model Can be used to develop explanatory theory Purpose of the additional qualitative synthesis Thematic analysis without theory generation Meta-aggregation Realist review EPPI approach (thematic) Narrative synthesis Meta-ethnography Thematic analysis with theory generation Grounded theory Product Explanatory theory, analytical or conceptual framework or interpretative framework/ mechanism Product Aggregated findings from source papers

30. Comparing characteristics of different approaches (developed for synthesis) Meta-ethnographyMeta-aggregation PurposeSeek and reveal similarities and differences, achieving a degree of innovation To aggregate findings of included studies and aid decision making EpistemologyIdealismRealism Quality AssessmentNot discussed, relevance argumentRequired, using standardized critical appraisal instrument SynthesisRefutational and reciprocal translation, line of argument synthesis Aggregation of findings into categories and of categories into synthesized findings OutcomeHigher order interpretation of findings, theory Standardised chart informing practice and policy

31. PART 3: Revent developments Context-specific versus multi-context reviews Mixed methods reviews

32. Context-specific versus multi-context syntheses Quantitative ReviewQualitative Review

33. Context-specific versus multi-context reviews Multi-context reviews Exhaustive search Little access to or knowledge of local databases and experts Targets a broad audience (but no-one in particular) Findings may be too general Risk of downplaying important local characteristics Context may get lost Potential low level of acceptance in end-users Wide ranging in scope Ability to cross compare different settings Works for topics were little heterogeneity between settings is expected Findings are more likely transferable to a broad range of settings Context-specific reviews Selective search (related to context) Access to and knowledge of local databases and experts Only relevant to the ‘happy few’. Findings are less likely transferable to other settings Targeted audience Highly relevant to practice and policy Maintains integrity with the context reported in original studies Findings may induce a higher level of acceptance in the end-users

34. Context-specific versus multi-context reviews Context-specific syntheses do well in responding to the needs and policies of a targeted setting. Multi-context syntheses assist in building a cumulative knowledge base and are an excellent choice when little heterogeneity is expected. Integrating the best of both – Umbrella reviews, in which insights in a particular phenomenon generated from different settings could be summarized. – Transcontextual adaptation, which means modifying insights in such a way that they become relevant and reply to the needs and policies of a targeted setting.

36. QES: conclusion If you wish to embark in the field of qualitative evidence synthesis you will soon realise that – There are no fixed standards – There is no general consensus (and will there ever be one?) – You shall have to travel slowly, – By your own means (but the CQRM-group can help), – On small and potentially difficult roads… BE PREPARED FOR A POTENTIAL DELAY!