Presentation is loading. Please wait.

Presentation is loading. Please wait.

Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES) Andrew Booth, Chris Carroll & Janet Harris, School.

Similar presentations


Presentation on theme: "Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES) Andrew Booth, Chris Carroll & Janet Harris, School."— Presentation transcript:

1 Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES) Andrew Booth, Chris Carroll & Janet Harris, School of Health and Related Research (ScHARR), University of Sheffield.

2 By the end of this workshop participants will be able to: Understand role of qualitative evidence syntheses (QES) /systematic reviews and types of questions they can address. Be able to formulate an appropriate question to be addressed by a qualitative systematic review/evidence synthesis. Be able to construct an efficient search strategy for retrieval of primary qualitative research and to select appropriate methodology filters for qualitative studies. Apply explicit criteria regarding collection/analysis of qualitative data to appraise the quality of a qualitative research report. Understand main features and techniques required to conduct QES, specifically thematic synthesis and framework synthesis. Understand primary considerations when selecting a synthesis method and how these impact upon the conduct and reporting of a qualitative evidence synthesis/systematic review.

3 Programme MORNING What is Qualitative Systematic Review/ Evidence Synthesis What are your choices? Formulating a Question for Qualitative Review Searching & Sifting for Relevant Studies Break Quality Assessment [Plenary] Quality Assessment Workshop

4 Programme AFTERNOON Introducing Thematic Synthesis/Framework Synthesis Thematic Synthesis/Framework Synthesis Workshop Feedback Break Producing Your Review for Publication Where to Next? Close

5 What is Qualitative Systematic Review/Evidence Synthesis? Andrew Booth School of Health and Related Research, University of Sheffield, UK

6 Umbrella terms: Qualitative Systematic Review (but also non-quantitative, e.g. not meta-analysis) Qualitative Metasynthesis (Sandelowski et al, 2001) Qualitative Research Synthesis (Howell Major & Savin-Baden, 2010) Qualitative Evidence Synthesis (Cochrane Collaboration: Hannes & Lockwood, 2012)

7 Topic=("qualitative systematic review" OR "qualitative evidence synthesis" OR "qualitative research synthesis") OR Topic=(metastudy OR metasynthesis OR "meta synthesis" OR "meta ethnography" OR "meta ethnographic" OR "metaethnography" OR "metaethnographic") OR Topic=("systematic review of qualitative") = 645 Currently about 10 per month 61% from ; 37.5% from

8 So what is a QES? qualitative evidence synthesis – an umbrella term increasingly used to describe a group of review types that attempt to synthesise and analyse findings from primary qualitative research studies (Booth et al, 2011).

9 Qualitative Evidence Synthesis Quantitative meta-synthesis, or meta-analysis, aims to pool numerical results of individual quantitative studies, qualitative meta-synthesis looks for themes or constructs that lie in or across individual qualitative studies. Within broader category of qualitative meta- synthesis, (Cochrane preferred term = qualitative evidence synthesis) narrow term meta-ethnography refers to specific method of data synthesis most widely adopted to date. Booth et al, 2006

10 Key Difference Goal not aggregative in sense of adding studies together, as with meta-analysis. Interpretative in broadening understanding of a particular phenomenon.

11 An Example Paterson and colleagues identified 38 studies examining first hand experience of living with diabetes. Prevailing metaphor = concept of balance. Specific sub-themes identified across multiple studies included knowing one's body, learning how to manage diabetes, and fostering supportive, collaborative relationships with others.

12 Stages of a Qualitative Evidence Synthesis? Formulating the review question Conducting a systematic literature search Screening and selecting appropriate research articles Analyzing and synthesizing qualitative findings Maintaining quality control Presenting findings (Sandelowski & Barroso, 2007)

13 Framework for systematic reviews of qualitative research (Garside, 2010) StageTypical activities Developing research questionAssemble team; Consult; Agree approach Scoping exerciseIdentify relevant research; Refine methods Identifying relevant literatureDevelop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reportsPreliminary reading; Identify theories; Assess utility/relevance Analysis and synthesisReading & rereading study reports; Constant comparison; Assess validity Preliminary synthesisCategorising; tabulating; mindmaps; Explore relationships Full synthesisThematic analysis; translation of findings; Theory development; rival explanations DisseminationTarget audiences; Limitations of review ThroughoutMultiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

14 Why do it? Perspectives of Patients, Relatives or Carers Of Condition Of Intervention Perspectives of Staff Implementation Barriers and Facilitators Patient Centred Outcomes Development and Testing of Theory Answering What Works for Whom Under Which Circumstances? Acceptability of Treatments Explaining Differential Effectiveness Designing and Evaluating Complex Interventions

15 Some Examples - 1 Perceptions of patients and physiotherapists on patient participation Systematic search in six databases using a set of key words, extracted relevant data, performed quality assessment and synthesized findings from selected studies. Narrative synthesis of qualitative studies. Retained 11/160 studies. Two main themes: conceptualization of patient participation and patients' role preferences. Patient participation included goal setting, information exchange, decision-making and exercise training; often influenced power relation between patient and physiotherapist. Patients' willingness to participate varied; they often did not play their desired role. SCHOEB & BÜRGE (2012)

16 Some Examples - 2 Incentives and barriers to lifestyle interventions for people with severe mental illness. Eight electronic databases [1985–March 2009] plus Cochrane Library and Google Scholar. Electronic hand searches of key journals and explosion of references also undertaken. Narrative synthesis of qualitative, quantitative and mixed methods studies Reports possible incentives and barriers including: Illness symptoms, treatment effects, lack of support and negative staff attitudes as possible barriers; Symptom reduction, peer and staff support, knowledge, personal attributes and participation of staff as possible incentives. ROBERTS & BAILEY (2011)

17 Centres and Networks: EPPI-Centre Joanna Briggs Institute Cochrane Collaboration Campbell Collaboration Software EppiReviewer QARI Books Noblit & Hare (1988) - Meta-Ethnography Patterson et al (2001) - Meta-Study of Qualitative Health Research Sandelowski et al (2001) - Handbook for Synthesizing Qualitative Research Petticrew & Roberts (2003) - Systematic Reviews in Social Sciences Mays, Pope, Popay (2007) - Synthesizing Qualitative and Quantitative Health Research Howell Major & Savin-Baden (2010) - Introduction to Qualitative Research Synthesis Hannes & Lockwood (2011) - Synthesizing Qualitative Research: Choosing the Right Approach Saini & Schlonsky (2012) - Systematic Synthesis of Qualitative Research Booth et al (2011) – Systematic Approaches to a Successful Literature Review Gough, Oliver & Thomas (2012) - An Introduction to Systematic Reviews

18 What are your Choices? Andrew Booth School of Health and Related Research, University of Sheffield, UK

19 Specific Methods Realist Synthesis Meta-Interpretation Critical Interpretive Synthesis Thematic Synthesis Framework Synthesis Narrative Synthesis Meta-Ethnography Best Fit Synthesis Meta Study Meta Narrative EPPI- Centre Method Matrix Method Bayesian Meta- Synthesis Meta- Aggregation

20

21 Decision to conduct a qualitative evidence synthesis To interpret synthesised qualitative evidence and develop explanatory theory or models Purpose of the additional qualitative synthesis or Choice of Synthesis (Adapted from Noyes & Lewin, 2011) To aggregate/ summarise/ integrate qualitative data to address questions in relation to a specific intervention review Primarily to integrate and interpret qualitative and quantitative evidence within a single approach or integrated model. Can be used to develop explanatory theory. Thematic analysis without theory generation Meta-aggregation Meta-summary Product: Aggregated findings from source papers Meta- ethnography Grounded theory Thematic analysis with theory generation Framework synthesis Realist Review EPPI Approach Matrix Method Narrative Synthesis Bayesian Synthesis Critical Interpretive synthesis Product: Explanatory theory, analytical or conceptual framework or interpretative framework/mechanism Best fit synthesis

22 Bad Reasons for Choosing Method Frequency of Use of Method (e.g. Meta- Ethnography) Popularity/Sexiness of Method (e.g. Realist Synthesis) What a friend/ colleague/ mentor has used (once!) Bad experiences of others (may have been inappropriate!)

23 Key Considerations 1.Role of Theory 2.Available Expertise 3.Extent of Description versus Interpretation 4.Intended Product Other Considerations: Richness of Data and Available Time/ Resources

24 1. Will You Generate, Explore, Test Theory (Gough et al, 2012)? Generate – may require suspension of disbelief – quality assessment/ value judgement may come later (cp. Brainstorming)- Grounded Theory, Meta-ethnography Explore – looking for patterns - Narrative Synthesis, Thematic Synthesis Test – quality assessment differentiates well- supported and unsupported data - Framework Synthesis (incl. Best Fit Synthesis) NB. We (Carroll & Booth) are currently conducting empirical work on systematic identification of Theories

25 2. What Expertise Can You Access? Epistemological versus Technical: Expertise in Qualitative Research Methods (e.g. Grounded Theory; Framework Analysis, Thematic Analysis) Expertise in Synthesis Methods (incl. Searching, Data Extraction, Quality Assessment, Interpretation) Knowledge of Topic Area

26 3. Will You Describe or Interpret? All Reviews figure on a continuum between Description and Interpretation Description – What does the data say? – factual reporting of studies, themes etc… Reader does work of interpretation Interpretation – What does the data mean? –subjective interpretation of data and themes etc… Reviewer does work of interpretation – may be contested

27 4. What is Your Intended Output/Product? the output of some methods of synthesis (Thematic Synthesis, textual Narrative Synthesis, Framework Synthesis, and ecological triangulation) is more directly relevant to policymakers and designers of interventions than the outputs of methods with a more constructivist orientation (Meta-Study, Meta-Narrative, Meta- Ethnography, Grounded Theory, CIS) which are generally more complex and conceptual (Barnett-Page & Thomas, 2009) Thematic Synthesis (including Meta-Aggregation) and Framework Synthesis produce findings to directly inform practitioners (Thomas & Harden, 2009) Interpretive approaches (e.g. CIS, Meta-Ethnography) produce a model that requires practitioners to interpret relevance and applicability to their own context Narrative Synthesis or EPPI- Centre (matrix) methods may help to integrate and present quantitative/qualitative work

28 How Rich (Thick) is Your Data? Qualitative data from thin studies (or textual responses to surveys) will not sustain interpretive approaches Limited to Meta- Aggregation, Thematic Synthesis, Framework Synthesis, Narrative Synthesis –type approaches Rich/Thick reports will sustain Meta- Ethnography/Grounded Theory – may allow selective sampling/ theoretical saturation NB. Is Unit of Analysis Individual Study (Meta- Aggregation, Thematic Synthesis) or Body of Evidence (e.g. Meta-Narrative or Critical Interpretive Synthesis approaches) or even Theory (Framework Synthesis/Best Fit Synthesis)?

29 How Long?/How Much Have You Got? Richer approaches make fuller use of data – require fewer studies Meta-Aggregation, Thematic Synthesis can handle large numbers of studies Framework/best fit approach for speed 32 papers (775 patients and carers) reporting help-seeking experiences for at least 20 different types of cancer.

30 Synthesis Quiz For each of the following scenarios identify the review characteristics and try to match to an appropriate type of synthesis

31 Scenario A: You are working in a general health technology assessment team conducting a synthesis of Group Therapy for Postnatal Depression. There are few studies and most reports are Nursing Standard/Times case reports. There is no prevalent theory. You will produce recommendations on what works for whom. 1.Role of Theory2. Expertise 3. Describe/Interpret 4. Output/Product

32 Scenario B: You are working as a group of topic experts and experienced qualitative researchers to examine the phenomenon of Willingness to Hasten Death. There are less than eight rich qualitative studies. No-one has yet conducted a synthesis to look at what is meant by the concept. Your review will help those who work in terminal care to gain a better understanding of the phenomenon. 1.Role of Theory2. Expertise 3. Describe/Interpret 4. Output/Product

33 Scenario C: You are a member of a government- funded Institute producing best practice reviews for nurses and nursing managers. You are asked to examine all evidence, quantitative and qualitative, for physical restraint in residential homes. You are expected to turn around a report in a very short time frame. Your final review is expected to include Recommendations for Practice. 1.Role of Theory2. Expertise 3. Describe/Interpret 4. Output/Product

34 Scenario D: Your review team, with access to clinical expertise, is commissioned to review the evidence on taking vitamins and supplements to prevent colorectal cancer. Because of the focus on long term outcomes there are few directly relevant qualitative studies. You decide to look for related studies and hypothesise attitudes to these specific agents from a wider evidence base. Attitudes to medicine are well-theorised but not within such a long-term prevention perspective. Your qualitative review will be a short chapter in a much longer effectiveness/cost effectiveness report. 1.Role of Theory2. Expertise 3. Describe/Interpret 4. Output/Product

35 Safest Options! If… There is a Pre- existing Theory or Framework…. Then ….Framework Synthesis (including Best Fit Synthesis) If… There is No Theory or Framework… Then …Thematic Synthesis (Can also act as first stage of Meta- Ethnography) If… There is a Proximate (Close-ish!) Theory or Framework…. Then ….Best Fit Synthesis

36 Formulating a Question for Qualitative Review Janet Harris School of Health and Related Research, University of Sheffield, UK

37 Framework for systematic reviews of qualitative research (Garside, 2010) StageTypical activities Developing research questionAssemble team; Consult; Agree approach Scoping exerciseIdentify relevant research; Refine methods Identifying relevant literatureDevelop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reportsPreliminary reading; Identify theories; Assess utility/relevance Analysis and synthesisReading & rereading study reports; Constant comparison; Assess validity Preliminary synthesisCategorising; tabulating; mindmaps; Explore relationships Full synthesisThematic analysis; translation of findings; Theory development; rival explanations DisseminationTarget audiences; Limitations of review ThroughoutMultiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

38 A 15 minute guide What is the focus? Brainstorm questions Refine Prioritize Determine next steps Reflect (Adapted from Rothstein and Santana (2011) Teaching Students to Ask Their Own Questions: One small change can yield big results. Harvard Educational Letter)

39 Focus and brainstorm Our topic of interest today is: Taking a client-centred approach to rehabilitation after stroke Our FOCUS – the problem that has initiated a review of qualitative research : Health professionals have different ideas about client-centred approaches to rehabilitation Our AIM: To produce evidence from a synthesis of the literature that will guide effective client centred-approaches BRAINSTORM Ask as many questions as you can about this topic Do not stop to discuss, judge, or answer the questions. Write down every question exactly as it is stated. Change any statement into a question

40 What did you come up with? Broad or narrow question? Open or closed? Advantages/disadvantages of the questions?

41 Prioritize Which questions do you want to pursue and why? Consider Likelihood of finding qualitative studies Relevance of studies to problem focus How the research synthesis will be used What people have researched What I think I am looking for

42 Next steps would be… Conduct a scoping search to see whether the published research seems to be a match for your problem focus Use the brief scoping of abstracts to refine the question further What I think I am looking for What I can realistically find!!

43 The final question for the qualitative review was: What are stroke survivors experiences with rehabilitation? (Peoples et al, 2011)

44 Searching and Sifting for Relevant Studies Andrew Booth School of Health and Related Research, University of Sheffield, UK

45 Framework for systematic reviews of qualitative research (Garside, 2010) StageTypical activities Developing research questionAssemble team; Consult; Agree approach Scoping exerciseIdentify relevant research; Refine methods Identifying relevant literatureDevelop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reportsPreliminary reading; Identify theories; Assess utility/relevance Analysis and synthesisReading & rereading study reports; Constant comparison; Assess validity Preliminary synthesisCategorising; tabulating; mindmaps; Explore relationships Full synthesisThematic analysis; translation of findings; Theory development; rival explanations DisseminationTarget audiences; Limitations of review ThroughoutMultiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

46 Relevant Studies… 1.Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors), AND 2.Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]

47 Other Considerations… How much literature is there? How is it characterised? e.g. Was it conceived as Qualitative Research? How far back will I go? Has the Intervention/Experience of Living with The Condition Changed? What Settings, Countries, Languages will I include? How important is Context?

48 Post acute stroke rehabilitation

49

50 Relevant Studies… 1.Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors),

51 Matching the Focused Question Combination of natural language (e.g. stroke) and index terms (e.g. CEREBROVASCULAR DISORDERS and BRAIN INFARCTION) CEREBROVASCULAR DISORDERS BRAIN INFARCTION May include subheadings e.g. /rehabilitation as well as index terms Need to optimise Sensitivity (Getting everything you want) and Specificity (Getting only what you want)

52 Sample Search Strategy (SIGN Guideline)

53 Relevant Studies…, AND 2.Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]

54 Challenges Qualitative research is small part of the research literature Indexing in databases is inconsistent and variable Qualitative studies sometimes have creative titles or inadequate abstracts MEDLINE might not be your preferred resource. If searching in limited databases try MEDLINE and CINAHL

55 How would you find qualitative studies?

56 Sources of Evidence Bibliographic databases and catalogues Papers suggested by individuals Supplementary Methods Web and Grey literature searches 25/04/2014© The University of Sheffield

57

58

59 Identifying Qualitative Research - Terminology Generic terms: e.g. qualitative plus ESCAPADE Exploratory Methods: Focus group, Grounded theory, Action Research, Content analysis, Thematic analysis Software: Nudist or NVivo Citations: Glaser & Strauss Application: Ethnology, Psychology Phenomenon: Perceptions, Attitudes, User Views, Standpoint, Viewpoint Approaches: Ethnographic Data: Stories, Narratives, Descriptions, Themes, Findings Experiences: Encounters, Experiences

60 What is a methodological filter? A hedge or filter is a standardised search strategy that is designed to be used in conjunction with a subject search to retrieve valid studies from the (primary) medical literature. Filters work in one of two ways: by identifying particular publication types or study designs most likely to answer a question by isolating subject or free-text terms most likely to be associated with high-quality studies

61 How do they work? Filters come from 3 different sources:- Subject heading Keyword Publication type

62 How do I use a methodological filter? Step One: Carry out a subject search as usual using subject headings and/or free text Step Two: Apply methodological filter appropriate to question you are asking One-line filter Maximum sensitivity filter Maximum specificity filter Mid-range filter

63 One-line filters MeSH Headings e.g. Qualitative Research [Medline 2003-] Keyword Findings Publication Type Research [CINAHL only]

64 Methodological filters qualitative$ 2.findings 3.interview$ 4.interviews.DE. 5.1 OR 2 OR 3 OR 4 Grant MJ. How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Info Libr J Mar; 21(1):21-32.

65 Methodological filters - 2 Best sensitivity: exp interviews Best specificity: audiorecording.sh. Best optimization of sensitivity and specificity: exp study design Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res May;17(5):

66 Methodological filters - 3 Dont forget MeSH heading Qualitative Research Introduced in 2003, so coverage is limited But you can combine it with one of the previous filters by using the OR operator

67 Sources of filters ISSG search filters resource SIGN McMaster Hedges Project PubMed Clinical Queries

68 Each Question requires a different solution... Findings showed that a simple search strategy (broad-based terms - 3 search terms) was as effective as a complex one (free text - 48 search terms) in locating qualitative research on patients experiences of living with a leg ulcer. May be feasible to restrict searches with a clear nursing focus to CINAHL bibliographic database. Replication with other nursing topics is required. Flemming K, Briggs M. Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs Jan;57(1):

69 Supplementary Methods Citation searches Reference lists Hand searching 25/04/2014© The University of Sheffield

70 Citation searching- Google Scholar

71 In Conclusion Identifying Qualitative Research is Challenging Simple strategy with qualitative terms (e.g. Interview*; qualitative; findings etc) may be more effective than exhaustive lists of terms May not be necessary to search as many databases as quantitative topics But supplementary methods (e.g for books and theses) may be very important! Cp. Cluster searching!

72 Reading the Reports Janet Harris School of Health and Related Research, University of Sheffield, UK

73 Aims and objectives What do we mean by quality assessment? What does it involve? How do you do it? What are the strengths and weaknesses of different tools or checklists?

74 Framework for systematic reviews of qualitative research (Garside, 2010) StageTypical activities Developing research questionAssemble team; Consult; Agree approach Scoping exerciseIdentify relevant research; Refine methods Identifying relevant literatureDevelop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reportsPreliminary reading; Identify theories; Assess utility/relevance Analysis and synthesisReading & rereading study reports; Constant comparison; Assess validity Preliminary synthesisCategorising; tabulating; mindmaps; Explore relationships Full synthesisThematic analysis; translation of findings; Theory development; rival explanations DisseminationTarget audiences; Limitations of review ThroughoutMultiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

75 Quality assessment: Basic criteria Carrying out ethical research Importance of the research Clarity and coherence of the report Use of appropriate and rigorous methods Importance of reflexivity or attending to researcher bias Importance of establishing validity or credibility Importance of verification or reliability (Cohen et al, 2008)

76 Quality assessment: 3 stages 1.Filtering against minimum criteria, involving adequacy of reporting detail 2.Assessing technical quality of the studys elements - methodological soundness, credibility, validity 3.Judging theoretical consistency

77 Filtering: Relevance and adequacy of reporting Relevance: generally judged against title and abstract in the first instance Reporting: Is the study process reported well enough to judge technical rigour? This depends on your area of interest… Intervention Core elements of the intervention as planned Non-core elements e.g. components that are added to suit a particular context or/or population Theory development: Relationship between Context, Mechanisms and Outcomes Implementation: barriers and facilitators to implementing the intervention

78 Quality appraisal: Basic criteria Technical quality – Qualitative concepts Techniques Credibility: the representation of data fits the views of the participants studied, the findings hold true outside auditors or participants validate findings (member checks) peer debriefing, attention to negative cases, independent analysis of data by more than one researcher verbatim quotes persistent observation (stay in the field long enough) Confirmability: findings are qualitatively confirmable through the analysis being grounded in the data, through examination of the audit trail assessing the potential effects/impact of the researcher during all steps of the research process Reflexivity toward personal influences, bias providing background information on the researchers background, education, perspective, school of thought Dependability : process of research is logical, traceable and clearly documented, particularly on the methods chosen and the decisions made by the researchers peer review, debriefing, audit trails triangulation, the use of different methodological approaches to look at the topic of research reflexivity to keep a self-critical account of the research process calculation of inter-rater agreements Transferability : research findings are transferable to other specific settings providing details of the study participants to enable readers to evaluate for which target groups the findings potentially hold true providing contextual background information, demographics providing thick description about both the sending and the receiving context

79 Validity: linking data collection and analysis to conclusions When evaluating methodological soundness, we need to know whether the set of arguments or the conclusion derived from a study necessarily follows from the premises. whether it is well grounded in logic or truth. whether it accurately reflects the concepts, the ideas that it is intended to measure.

80 Theoretical quality Does your review question aim to develop an explanation for something – a theory? If it does, then you need to consider theoretical quality How was the theory developed? Is there congruence between the data and the interpretation of findings? Can you see a logical connection or translation of findings into theory?

81 Revisiting relevance? Relevance can be a multistage process The first stage involves excluding obviously irrelevant papers (filtering) - The second stage involves weighing relevance to the review question with the quality of reporting and technical competence at establishing validity Data may be relevant and insightful but not rigorously analyzed...do you include it?

82 Criteria 1) There is congruity between the stated philosophical perspective and the research methodology. 2) There is congruity between the research methodology and the research question or objectives. 3) There is congruity between the research methodology and the methods used to collect data. 4) There is congruity between the research methodology and the representation and analysis of data. 5) There is congruity between the research methodology and the interpretation of results. 6) There is a statement locating the researcher culturally or theoretically. 7) The influence of the researcher on the research, and vice-versa, is addressed. 8) Participants, and their voices, are adequately represented. 9) The research is ethical according to current criteria or, for recent studies, there is evidence of ethical approval by an appropriate body. 10) Conclusions drawn in the research report do appear to flow from the analysis, or interpretation, of the data. TOTAL QARI Critical Appraisal Instrument

83 Criteria categorization and definition Study design: Yes - if e.g., a case study approach was used..., phenomenology was used... No - If paper does not specify study design Unclear if unsure Method of data collection: Yes - If details of the data collection method are given e.g., piloting; interviews, topic guides for interviews; number of items in a survey; use of open or closed items; validation, and so forth. No - If only states focus group, interviews were used or questionnaire was used Unclear if unsure Selection of participants: Yes - If the sampling and recruitment of participants is described in full or explicitly as e.g., purposive, convenience, theoretical and so forth. No - If only details of participants are given, e.g. age, gender, number Unclear if unsure Methods of data analysis: Yes - If full details of analysis method are given, e.g., transcription and form of analysis (with reference or full description of method), validation tests, and so forth. No - If only states content analysis or that data were analyzed Unclear if unsure Quality of Reporting Tool (Carroll et al, QHR 2012)

84 Appraising a paper for a qualitative review Group 1: Use the QARI appraisal tool Group 2: Use the Quality of Reporting Tool Make your decision about the quality of the paper Technical quality Relevance Theoretical quality

85 Integrating papers Would you use this paper in the qualitative review? Poor methodological quality but relevant data and valuable insights Good quality, but poor interpretation and limited insight Which gets privileged – the quality score or useful insight?

86 Learning outcomes An understanding of What we mean by quality assessment What critical appraisal involves How you conduct an appraisal The strengths and weaknesses of different tools or checklists Which tool might be best for you

87 Introducing framework synthesis and thematic synthesis Chris Carroll School of Health and Related Research, University of Sheffield, UK

88 Aims and objectives What do we mean by framework synthesis and thematic synthesis? How do you extract data for synthesis? How do you perform both types of synthesis? How do you complete the synthesis?

89 TimeContent Introduction to two types of synthesis Extracting and coding data: Introduction Practical (10 mins) Feedback (5 mins) How to do thematic analysis Practical (15 mins) Feedback (10 mins) Analysis and synthesis Feedback Producing your review for publication (AB)

90 Framework for systematic reviews of qualitative research (Garside, 2010) StageTypical activities Developing research questionAssemble team; Consult; Agree approach Scoping exerciseIdentify relevant research; Refine methods Identifying relevant literatureDevelop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reportsPreliminary reading; Identify theories; Assess utility/relevance Analysis and synthesisReading & rereading study reports; Constant comparison; Assess validity Preliminary synthesisCategorising; tabulating; mindmaps; Explore relationships Full synthesisThematic analysis; translation of findings; Theory development; rival explanations DisseminationTarget audiences; Limitations of review ThroughoutMultiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

91

92 A Framework

93 How do you create a framework? Develop own framework or themes or concepts for coding Oliver S et al: A multidimensional conceptual framework for analysing public involvement in health services research. Health Expectations 2008, 11: Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing Childrens, Young Peoples and Parents Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; Identify or create a thematic framework or conceptual model Carroll C, Booth A, Cooper K. A worked example of best-fit framework synthesis: A systematic review of views concerning the taking of potential chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29

94 An existing conceptual framework Conrad K, et al (1996). The worksite environment as a cue to smoking reduction. Research in Nursing & Health,

95

96 A Framework

97 Framework synthesis Thematic synthesis

98 The case study The question: What are the barriers and facilitators of patient autonomy post stroke ? The studies: 1.Proot I et al, Patient autonomy during rehabilitation: the experiences of stroke patients in nursing homes, International Journal of Nursing Studies, 2000; 37: Mangset M et al, Were just sick people, nothing else:... Factors contributing to elderly stroke patients satisfaction with rehabilitation, Clinical Rehabilitation, 2008; 22:

99 Data extraction What is it? An attempt to reduce a mass of material (your included papers) to a much smaller body of text and numbers, amenable to analysis and the interpretation of findings

100 Data extraction form Location Setting Sample (n) Age, Gender, Ethnicity and other relevant variables (e.g. Marital status, socio-economic status) Intervention (if any) Outcomes and results Study design Data collection method Quality assessment criteria Further citations Format?????

101 Example form: Proot 2000

102 What results do you extract? What is your question? Question: What are the barriers and facilitators of patient autonomy post-stroke? Keep the question in mind as you read: Are the data relevant to this question? Is the question answered by the data?

103 Extraction of results Results sections of included studies: Authors statements clearly-supported by data and/or Verbatim quotations... It was often difficult to distinguish [Schutzs] first- from second-order constructs Atkins et al. Conducting a meta-ethnography of qualitative literature: Lessons learnt, BMC Med Res Methodol 2008; 8: 21 Schutz A. Collected Papers 1971; vol.1: 361.

104 From: Mangset p.829

105 Extracting data for thematic analysis

106

107 Extracting or coding data using framework analysis Framework developed from... Kirkevold M et al. Promoting psychosocial well-being following a stroke: developing a theoretically and empirically sound complex intervention, International Journal of Nursing Studies, 2012; 49:

108 Extracting or coding data using framework analysis Code numberThemeDefinition 1DignityA self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself 2Well-beingA basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness 3CopingAbility to adapt and cope within home and hospital settings 4Family SupportRelations with immediate family 5Support for patients coping efforts Patients take responsibility and are supported by others in this 6Bodily changes and impairments Difficulties generated by residual or post- stroke conditions Handout 5

109 Extracting or coding data using framework analysis

110

111

112

113 Extraction and coding: Practical Consider the full framework in Handout 5 Consider extracts A, B and C on Handout 6 Code the extracted data according to the framework by entering the number of the framework code in the right- hand column on Handout 6 Note: Data may be assigned more than 1 theme No code in the framework, in your opinion, might capture, fit or reflect the data. If this is the case, enter New theme into the right-hand column 10 minutes: Individual and compare with neighbour; 5 minutes: Discussion of results and experiences

114 What now? Secondary thematic analysis Practical: Thematic analysis Discussion of practical Synthesising your data

115 What now? You have extracted your data AND You have coded your data against your framework (framework analysis) Or you may have coded some of your data but not others (best fit framework synthesis) OR You are conducting thematic synthesis anyway... You now need to assign themes to those data without themes, i.e. Using secondary thematic analysis (data reduction, display, conclusion) How do you do it?

116 Generating new themes Evidence does not speak for itself. It requires interpretation.... on the basis of our personal experience and a range of ideas that we use to make sense of our observations Kelly M et al. Evidence-based public health: A review of the experience of NICE developing public health guidance in England, Social Science and Medicine, 2010; 71:

117 Developing new themes C. …I will take that paper when P. is finished with it. I want to study it. Then I'll take my shoes off, and practise as I see it in the picture (p04b, p. 11), Proot, p.271 Other / New theme? Interpret Create Revisit and revise Discuss Agree... = Learning self-help skills / guided self- determination

118 Thematic analysis: Practical Consider extracts D, E and F on Handout 6 Develop and assign new themes to these extracts and enter them in the right-hand column on Handout 6 Note: Data may be assigned more than 1 theme There may also be code in the framework which, in your opinion, might capture, fit or reflect the data. If this is the case, also enter this code into the right-hand column 15 minutes: Individual and compare with neighbour; 10 minutes: Discussion of results and experiences

119 Generating new themes: Practical Feedback Interpretation Bias? Reviewers might have agendas too What is the result of the secondary thematic analysis?

120 Synthesis Report the results of the new thematic framework Narrative structured by theme Report each theme with reference to studies and data in order to: Specify the evidence base for the theme Illustrate the theme with reference to actual data McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):

121 Synthesis

122 Reduce the findings into a smaller number of categories and then a single finding (metasynthesis, according to the Joanna Briggs Institute approach): McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):

123 Synthesis

124 A new framework Code numberThemeDefinition 1Dignity A self-concept characterised by self-valuing, self- acceptance, usefulness and belief in oneself 2Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness 3Coping Ability to adapt and cope within home and hospital settings 4Family Support Relations with immediate family 5Support for patients coping efforts Patients take responsibility and are supported by others in this 6Bodily changes and impairments Difficulties generated by residual or post-stroke conditions 7Self-determination Define theme as a concept, based on data 8Need for information Define theme as a concept, based on data 9Lack of shared decision- making Define theme as a concept, based on data 10Acceptance of therapy Define theme as a concept, based on data

125 A new framework and synthesis ThemeCategories Dignity Psychological factors Well-being Self-determination Coping Bodily changes and impairments Physical factors Need for information Interaction with professionals and family Lack of shared decision-making Family Support Support for patients coping efforts Acceptance of therapy

126 Synthesis Alternatively, go beyond the framework to create a new conceptual model or theory... Revisit data to explore the relationships between the themes or findings of your framework The richer or thicker the data, the deeper the resulting model or theory Narrative and diagrammatic representations

127 Synthesis

128 Do you stop there? Test your synthesis? Disconfirming cases Booth A, Carroll C, Illott I. Desperately Seeking Dissonance: Identifying the Disconfirming Case, Qualitative Evidence Synthesis, Qualitative Health Research 2012 (in press). Morse, J. M. The significance of saturation. Qualitative Health Research 1995; 5, Sensitivity analysis By quality, population, location, setting etc. By frequency and thickness Carroll C, Booth A, Lloyd-Jones M. Should we Exclude Inadequately-reported Studies from Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews, Qualitative Health Research 2012; 22(10) Boeije et al (2011) Making a difference: towards a method for weighing the evidence in a qualitative synthesis, Journal of Evaluation in Clinical Practice (4):

129 What is the purpose of your review and synthesis? To answer questions and to solve problems that cannot be addressed by quantitative research To illuminate patient and client experience and to develop theory To be useful

130 Learning outcomes An understanding of What we mean by framework synthesis What we mean by thematic analysis How you conduct framework and thematic synthesis, or best-fit framework synthesis The strengths and weaknesses of the different methods How you complete your synthesis

131 Summary Extract and interpret data Synthesis Test your synthesis Integrating and writing-up...

132 Producing Your Review for Publication Andrew Booth School of Health and Related Research, University of Sheffield, UK

133 Framework for systematic reviews of qualitative research (Garside, 2010) StageTypical activities Developing research questionAssemble team; Consult; Agree approach Scoping exerciseIdentify relevant research; Refine methods Identifying relevant literatureDevelop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reportsPreliminary reading; Identify theories; Assess utility/relevance Analysis and synthesisReading & rereading study reports; Constant comparison; Assess validity Preliminary synthesisCategorising; tabulating; mindmaps; Explore relationships Full synthesisThematic analysis; translation of findings; Theory development; rival explanations DisseminationTarget audiences; Limitations of review ThroughoutMultiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

134 Between Two Traditions Systematic Review Publishing Guidelines (e.g. PRISMA – formerly QUOROM) Primary Qualitative Research Reporting Traditions Qualitative Research Systematic Reviews QES

135 Requirements for a Published QES Systematic Review requires: Explicit methods Transparency Audit Trail Review Question Search strategies & sources Quality Assessment Method of Synthesis Strategies to reduce bias Primary Qualitative Research requires: Believability Findings Grounded in the Data Themes/Constructs Selective Findings Reflexivity

136

137 Purpose of Study Databases and Keywords Search Strategy Methods of Appraisal/ Extraction Methods of Synthesis

138

139 PRISMA Flow Diagram

140 NB. This is non- standard but informed by QUOROM/ PRISMA

141 Themes Findings Studies

142 Narrative synthesis Verbatim Extract Unsubstantiated Comment

143 Relationship between Themes

144 Structuring Recommendations

145 Where to Next? Andrew Booth School of Health and Related Research, University of Sheffield, UK

146 Further Reading Howell Major & Savin- Baden (2010) Hannes & Lockwood (2011)

147 Further Resources /

148 Further Support

149 Further Training Look out for SyNTAQuES – II (the Intermediate Version) from the Higher Education Academy ac.uk/events Advanced Training is available from: ESQUIRE13, September 2 nd -4 th 2013, ScHARR, University of Sheffield /scharr/shortcourseunit Previous materials at: pbworks.com

150 Some Issues to Think About! Or its not all that simple really!

151 Qualitative Systematic Review SystematicReviewQualitative Data Explicit? (cp. Iterative) Comprehensive?Qualitative Research? Reproducible? (cp. Reflexivity/ Subjectivity/ Interpretation) (cp. Theoretical Saturation; Purposive Sampling) Particular Type of Study (e.g. all Grounded Theory Studies)? Follows System: from Systematic Review or from Primary Qualitative Research? With/Without Quantitative Research? Qualitative Data?


Download ppt "Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES) Andrew Booth, Chris Carroll & Janet Harris, School."

Similar presentations


Ads by Google