(1) Systematic Reviews: The Potential of Different Methods Steve Higgins Henry Potts James Thomas Geoff Wong ESRC Research Methods Festival Oxford, 5 July 2012

(2) Introduction Aims –To outline major approaches to systematic reviewing and research synthesis –To demystify what on first glance looks like a bewildering set of overlapping approaches –To examine some of the assumptions that underpin these approaches –To discuss how these approaches might help in my (your!) situation

(3) Introduction Session structure: –20 Minute presentation + 10 minutes discussion / clarification of four approaches to synthesis –Coffee break after the 3 rd / 4 th presentation –Discussion in pairs –Questions + panel discussion

(4) What is a systematic review? Systematic: ‘done or acting according to a fixed plan or system; methodical’ Review: ‘a critical appraisal of a book, play, or other work’ (OED)

(5) What is systematic research synthesis? Synthesis : ‘The process or result of building up separate elements, especially ideas, into a connected whole, especially a theory or system’ (OED) Not just a report of the findings of the individual studies in a review Involves a transformation of the data from primary studies in some way Synthesis of the findings of all the included studies in order to answer the review question

(6) Aims of synthesis reviews Answer questions: what do we want to know and how can we know it? Bring together and ‘pool’ the findings of primary research (i.e. clarify what we know) Any question so potentially any type of evidence Driven by review users to answer relevant questions in relevant ways

(7) Introduction We will look at four main approaches to reviewing: –Meta-analysis –The synthesis of ‘qualitative’ research –Realist review –Meta narrative review Each have different disciplinary backgrounds and underlying assumptions They all bring together research using two main logics -

(8) Aggregation in reviews Aggregation refers to ‘adding up’ (aggregating) findings from primary studies to answer a review question… … to indicate the direction or size of effect … and our degree of confidence in that finding

(9) Configuration in reviews Configuration involves the arrangement (configuration) of the findings of primary studies to answer the review question…. … to offer a meaningful picture of what research is telling us... across a potentially wide area of research

(10) Most reviews contain elements of both, but some patterns are common AggregationConfiguration Approach to theoryTestGenerate Research questionClosedOpen ConceptsSecure prior to reviewEmergent in review Timing of methods decisions A prioriIterative / emergent Search for studiesRepresentative / unbiased / exhaustive Sufficient / purposive Quality appraisalAvoiding biasRichness of data Explore

(11) The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London, UK 1 Methods for the synthesis of qualitative research James Thomas 1 ESRC Research Methods Festival Oxford, 5 July 2012

(12) Synthesis of qualitative research: outline What can qualitative research tell us? Current developments/approaches in qualitative review methods Examples of syntheses of qualitative research

(13) What can qualitative research tell us?

(14) Qualitative research… aims to provide an in-depth understanding of people’s experiences, perspectives and histories in the context of their personal circumstances or settings…* with the use of unstructured methods which are sensitive to the social context of the study* the capture of data which are detailed, rich and complex…* is generalisable on the basis of theory not statistical probability ** * Spencer et al (2003); ** Popay (2006)

(15) Methods used include*… exploratory interviews focus groups observation conversation discourse and narrative analysis documentary and video analysis *Spencer et al (2003)

(16) Is it simply the analysis of textual data? “Increasingly, the terms ‘quantitative research’ and ‘qualitative research’ came to signify much more than ways of gathering data; they came to denote divergent assumptions about the nature and purposes of research in the social sciences” Bryman (1988) Quantity and Quality in Social Research. London: Unwin Hyman. p.3.

(17) Qualitative research can* help to define policy questions look in detail at how a programme or trial was actually implemented help to determine appropriate outcome measures by looking at ‘subjective’ outcomes help to clarify what counts as effective or successful * Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36

(18) …continued identify and explore unintended consequences contribute to service delivery and policy development by describing processes and contexts inform and illuminate quantitative studies, e.g. by contributing to the design of structured instruments, assessing the fairness of comparisons in experimental studies, or unpacking variation within aggregated data Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36

(19) Current developments in synthesising qualitative research Growing recognition of the value of qualitative research to inform policy & practice Methods developing to facilitate this

(20) Try:

(21) Methodological study We searched for methodological papers concerning the synthesis of concepts or theories –Purposive, rather than systematic search –Reference ‘chasing’ –Google scholar search –Handsearching key journals 203 papers retrieved 9 distinct methods for synthesis

(22) Methods for the synthesis of qualitative research identified Meta narrative synthesis critical interpretive synthesis meta-study meta-ethnography grounded theory thematic synthesis framework synthesis ecological triangulation

(23) Many methods: similar or different? Examined the methods across different dimensions: –Epistemology –Approach to quality assessment –Attitudes towards problematizing the literature –Use of review question –How similar / different the included studies were –Characteristics of the synthetic product Found they fell into two broad camps: ‘idealist’ and ‘realist’

(24) ‘Realist’ approaches Purpose To answer a policy relevant question Methods Qualitative/ quantitative data analysed with qualitative/ quantitative methods Searching linear or iterative Quality assessment of study methods Product Directly applicable to policy and practice decisions

(25) ‘Idealist’ approaches Purpose To explore and construct concepts from the data For generating theory Methods Qualitative data analysed with qualitative methods Searching iterative Qualitative assessment of study content > method Product Complex, requiring further interpretation before being used for policy or practice

(26) Conclusions Operationally, many methods are very similar Underlying principles differ Product differs in terms of the amount of additional interpretation required. This may reflect… (Barnett-Page and Thomas (2009))

(27) Perspectives on how evidence informs the policy process IdealistRealist Knowledge-driven model; Problem-solving (policy driven) model* (either researcher disseminates research or policy-maker commissions / requests specific research) Less suitedMore suited Interactive model; Enlightenment model* (ongoing interactions between researcher and policymaker; ‘gradual sedimentation of insights…’) More suitedLess suited *Weiss (1979)

(28) Two examples All this is a bit abstract… what do systematic reviews of qualitative research look like? Two examples: –The ‘fine grain’ of a synthesis –‘Mixed methods’ synthesis

(29) Example 1: Informing policy about the ‘obesity epidemic’ What are children’s views about: the meanings of obesity or body size, shape or weight (including their perceptions of their own body size)? influences on body size? changes that may help them to achieve or maintain a healthy weight? Views = attitudes, opinions, beliefs, understanding or experiences As distinct from health/weight status, behaviour, factual knowledge Rees et al 2009

(30) ‘Thematic synthesis’ Similar to other methods of synthesising qualitative research (e.g. ‘meta-ethnography’) Source data = text (documents) Source material = conceptual Key method = translation Final product = interpretation (Thomas and Harden 2008)

(31) Stages of thematic synthesis Stages one and two: coding text and developing descriptive themes –Identifying the ‘findings’ –Line-by-line coding –Developing descriptive themes Stage three: generating analytical themes –In the light of the review question

(32) Screenshot – line-by-line coding

(33) Screenshot – descriptive codes diagram

(34) Example 2. adding insights to other existing reviews How might a systematic review of qualitative research support or add insight to a meta- analysis? Product of qualitative synthesis can form the conceptual framework within which heterogeneity can be explored in a meta-analysis

Searching, screening mapping and user involvement e.g. Statistical meta-analysis of trials a)Data extraction b)Quality assessment c)Effect sizes pooled d)Narrative synthesis Addresses sub RQ e.g. ‘which interventions are effective?’ e.g. Thematic synthesis of qualitative studies a)Data extraction b)Quality assessment c)Descriptive themes d)Analytical themes Addresses sub RQ e.g. ‘what are people’s perspectives and experiences ? Synthesis 1Synthesis 2 Synthesis 3 Driven by overall review question Integration of separate syntheses e.g. a)Matches, mis-matches and gaps b)Hypotheses generated in synthesis 2 tested amongst trials in synthesis 1 Review Question e.g. ‘What is known about the barriers to, and facilitators of, outcome X (e.g. physical activity) amongst population A (e.g. young people

(36) Cross study synthesis: Findings Children’s viewsOutcome evaluations Recommendation for interventions Good qualityOther Do not promote fruit and vegetables in the same way No soundly evaluated interventions No other interventions identified Brand fruit and vegetables as an ‘exciting’ or child- relevant product, as well as a ‘tasty’ one 5 soundly evaluated interventions identified 5 other interventions Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 5 soundly evaluated interventions identified 6 other interventions identified

(37) Cross study synthesis: an example of sub-group analysis Increase (standardised portions per day) in vegetable intake across trials Little or no emphasis on health messages

(38) This method of synthesis across study types: preserves the integrity of the findings of the different types of studies allows us to integrate ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives allows the exploration of heterogeneity in ways in which it would be difficult to imagine in advance –protects against ‘data dredging’

(39) Assumptions / theoretical models Mixed methods synthesis can draw on thinking relating to mixed methods in primary research In terms of a frequently used taxonomy of paradigm stances (Creswell 2011) e.g. –Incommensurability (cannot be mixed) –A-paradigmatic (can be mixed and matched in different ways) –Complementary strengths (not incompatible, but are different and should be kept separate) –Dialectic (paradigms are important in different ways leading to useful tensions & insights) –Alternative paradigm (‘mixed methods’ paradigm; foundation in e.g. pragmatism)

(40) Conclusions Qualitative research offers valuable insights that can inform policymaking Methods for synthesising qualitative research are still being developed Selection of a particular method depends on the type of answer required (and the means by which the review will inform policy)

(41) References Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9:59. doi: / ( 2288/9/59) 2288/9/59 Creswell J (2011) Mapping the Developing Landscape of Mixed Methods Research in Teddlie C, Tashakkori A: SAGE Handbook of Mixed Methods in Social & Behavioral Research. New York: Sage. Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic Reviews. London: Sage Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8:45 doi: / ( Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: (

(42) Systematic Reviews: The Potential of Different Methods Steve Higgins Henry Potts James Thomas Geoff Wong ESRC Research Methods Festival Oxford, 5 July 2012

(43) Summary HeterogeneityAggregation / configuration Meta-analysisA problem in fixed effect Allowance made for in random effects For ‘exploration’ Fixed & random effects models are pure aggregation Sub-group & meta- regression both aggregate & configure Synthesis of qualitative research Expected. Key challenge is judging when contexts / concepts are too dissimilar to be compared. Similar concepts are ‘aggregated’; most synthesis involves configuration Realist reviewNecessary to obtain different contexts etc Aggregates within CMO configurations; configures between them Meta narrative reviewNecessary to obtain diversity in paradigms Aggregation within paradigms; configuration between them

(44) Your turn Research question Heterogen- eity? Meta- analysis Qualitative synthesis Realist review Meta narrative review Research question 1 Research question 2