Presentation is loading. Please wait.

Presentation is loading. Please wait.

META-ANALYSIS AND SYSTEMATIC REVIEWS IN QUALITATIVE RESEARCH: MISSION IMPOSSIBLE? Qualitative Inquiry Group Seminar University of Toronto, March 27, 2009.

Similar presentations


Presentation on theme: "META-ANALYSIS AND SYSTEMATIC REVIEWS IN QUALITATIVE RESEARCH: MISSION IMPOSSIBLE? Qualitative Inquiry Group Seminar University of Toronto, March 27, 2009."— Presentation transcript:

1 META-ANALYSIS AND SYSTEMATIC REVIEWS IN QUALITATIVE RESEARCH: MISSION IMPOSSIBLE? Qualitative Inquiry Group Seminar University of Toronto, March 27, Ellen MacEachen (Institute for Work & Health, University of Toronto) Scott Reeves (Li Ka Shing Knowledge Institute, University of Toronto)

2 Overview – 4 parts 1. Where did systematic reviews come from & why do we do them? 2. Brief overview of the systematic review process 3. Cases that draw out systematic review issues  Case I (Scott)  Case II (Ellen) 4. Reflections on the conduct and usefulness of systematic reviews of qualitative studies 2

3 1. What are systematic reviews?  Positivistic roots / standardization  Synthesis of primary studies  Multiple check points / dual reviewers / transparency  Assess quality of evidence (only include ‘rigorous’ studies)  Inference through statistical analysis  Newer Qual types (meta-ethnography… ) 3

4 Why do SRs? Useful:  Knowledge translation (policymakers/ practitioners)  Initial entry into field (academics)  Help define field (academics)  Outline areas of future research (acad/policy)  Grant applications (academics) 4

5 2. Brief overview of the systematic review process  A walk through the ‘recipe’ followed for systematic reviews 5

6 6

7 Excluded at this level n = 4256 Studies considered for QA (qualitative, quantitative and mixed methods (n =609 ) Supplemental articles from reference list n = 63 Step 6: Finding Synthesis Excluded at this level n = 13 Excluded at this level n = 20 n = 19 Studies passing quality appraisal n = 14 Studies passing quality appraisal n = 5 Medline Total = 1620 EMBASE Total = 1683 CINAHL Total = 443 PsycINFO Total = 1174 Sociological Abstracts Total = 279 ASSIA Total = 74 ABI Inform Total = 381 EconLit Total = 58 Business Source Premier Total = 585 Other Total = 240 Content Experts’ Refs Total = 840 What databases will be searched? What keywords are used in the search? What is the scope of the search? Languages? Dates? What stakeholders are involved and at what stages? How will we find papers not listed on databases? What articles are included/excluded? Quality control process for what is being included/excluded How will we assess quality? Two reviewers per paper, consensus approach What data will we extract? Two reviewers per paper at this stage, consensus approach Step 1: Library Search Step 2: Study Relevance Step 3: Division of QNT/QL Studies Step 4: Quality Appraisal Step 5: Data Extraction

8 3. SR ‘cases’ Case I – Making decisions about how to construct systematic reviews: the early days Interprofessional education (Scott) Case II – Struggles with the paradigm of systematic reviews: later days Work & health in small businesses (Ellen) 8

9 Interprofessional education (IPE) Case I: Making decisions about how to construct systematic reviews: the early days 9

10 REVIEW QUESTION what kind of IPE, under what circumstances, produces what kind of outcomes? SETTING PARAMETERS Discussion/consultation/agreement: definitions, approaches, processes LITERATURE SEARCHES Retrieval & screening Quantitative studies: Quality assessment, data extraction, synthesis SYNTHESIS Different types of IPE and associated outcomes 10 Mixed method studies: Quality assessment, data extraction, synthesis Qualitative studies: Quality assessment, data extraction, synthesis

11 Context information  Policymaker demands for IPE evidence (early 1990s)  SR – political response  In dark about SR process – new activity  ‘Side line’ work (enthusiasm) 11

12 SR processes  Inclusive approach to review team – 9 members  Conceptually inclusive (implicit/explicit IPE)  Methodologically inclusive (qual/quan)  Open stance – seeking guidelines/standards  Pragmatic – trial & error  Abstraction – pre-determined categories 12

13 Thinking about Quality quality of study (appropriate design, sampling, recruitment, validity, reliability) quality of information (good contextual info, explicit rationale, clear research questions, clear results) (e.g. CASP, EPPI Centre, Popay et al 1998) 13

14 Large and small SRs  Big was good: 107 studies Practitioners/policymakers - larger numbers (width)  Small was better - 21 studies Researchers – small numbers (higher quality) 14

15 Some reflections  Lots of discussion/debate create parameters – first IPE review  Proceed with caution  One step forward, two back  Gradual movement: from inclusion to exclusion 15

16 Some reflections  Different types of qualitative research difficult to synthesize  Qual / Quan / mixed methods – more problems  Quality assessment – best effort (pragmatics)  SR team dynamics 16

17 Case II: Systematic review on work and health in small businesses (Ellen)  Struggles with the paradigm of systematic reviews  2 parts:  qualitative review  mixed method review 17

18 SUB-QUESTIONS Qualitative literature: How do SB workplace parties understand and enact processes related to OHS? Quantitative literature: Do OHS interventions in SBs have an effect on OHS? REVIEW QUESTION What understandings, processes and interventions influence OHS in SBs? LITERATURE SEARCH Retrieval, screening of T&As FOCI FOR IN-DEPTH REVIEW Qualitative studies: Quality assessment, data extraction, synthesis Quantitative studies: Quality assessment, data extraction, synthesis OVERALL SYNTHESIS Understandings, processes and interventions that influence OHS in SBs STAKEHOLDER CONSULTATION 18

19 Struggles in a Qualitative review  How to observe the broad focus that can be taken with qualitative research while also subjecting it to the ‘recipe’ of a SR process  The SR process is systematic, not exploratory.  Pre-defined parameters 19

20 Struggles in a Qualitative review  How to synthesize the findings & insights of Qualitative studies while also preserving their context and theory  The systematic review ‘extracts’ findings to answer a specific question.  What insights do we end up with? What do we gain/lose? 20

21 Struggles in a Qualitative review  What is being counted/considered in a qualitative systematic review?  Varying approaches to data extraction.  pre-set-findings categories?  grounded theory process? 21

22 Struggles in a Qualitative review  What counts as a qualitative study?  Studies can self-identify as “qualitative” but not use recognizable qualitative methods.  E.g. Participatory studies  Our criteria: Some qualitative data & some qualitative analysis of it 22

23 Struggles in a Qualitative review  How to handle interesting data from studies that don’t meet quality criteria?  “Nuggets” Pawson (2006).  Descriptions of process that are a ‘nugget of gold’ for the question being asked. 23

24 Struggles in a Mixed Method review  How to achieve a steady process for both Qualitative & Quantitative sides of the review  Quantitative team-- focused on definitions, outcomes  Qualitative team--no fixed categories for outcomes.  Qual & quan work happens at different times--affects possibilities for concurrent synergy. 24

25 Struggles in a Mixed Method review  Theoretical differences about ‘conflict of interest’ during the quality assessment process  Qualitative team--sensitive to social/power relations  heated issue, agreed to disagree. 25

26 Struggles in a Mixed Method review  Differences about ‘levels of evidence’  QN additive approach  E.g. "best evidence synthesis guidelines”.  “How much evidence is there that this has an effect?”  QL studies—not about effectiveness  Some additive approach; e.g. concepts in 3 studies = theme  No overall conclusions about strength of evidence 26

27 IV. Reflections on the conduct and usefulness of systematic reviews  SR process developed from Quantitative paradigm; we are trying to adapt Qualitative literature to it… …Can this be meaningfully done? 27

28 Issues Reviews are not entirely systematic  Messiness  Paired reviewers: anti-bias ‘ideal’  Politics of agreement/consensus BUT:  Power relations among reviewers  Assessing papers: own merits or against overall standard?  QA approaches change over course of the review 28

29 Issues  The issue of the podium  Arrogance: decisions on inclusion of peer reviewed papers  A side effect of systematic reviews  Claims of ‘ownership’ of a field from SR work 29

30 Issues  Debates:  Can the synthesis integrate research using different methods/theories  Although there are multiple descriptions/ explanations of data, these all ultimately relate to some underlying reality/truth (Bondas & Hall 2007, Mays et al 2005) 30

31 Some gains  Think carefully about papers  Learn a lot about a field  Read a lot of poor papers (learn: good, bad ugly)  Hone critical appraisal skills (teaching, journal reviewing/editing)  Opportunities to engage with people (outside academia) 31

32 Thank you  Questions? Comments? 32


Download ppt "META-ANALYSIS AND SYSTEMATIC REVIEWS IN QUALITATIVE RESEARCH: MISSION IMPOSSIBLE? Qualitative Inquiry Group Seminar University of Toronto, March 27, 2009."

Similar presentations


Ads by Google