Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES) Andrew Booth, Chris Carroll & Janet Harris, School of Health and Related Research (ScHARR), University of Sheffield.

By the end of this workshop participants will be able to: Understand role of qualitative evidence syntheses (QES) /systematic reviews and types of questions they can address. Be able to formulate an appropriate question to be addressed by a qualitative systematic review/evidence synthesis. Be able to construct an efficient search strategy for retrieval of primary qualitative research and to select appropriate methodology filters for qualitative studies. Apply explicit criteria regarding collection/analysis of qualitative data to appraise the quality of a qualitative research report. Understand main features and techniques required to conduct QES, specifically thematic synthesis and framework synthesis. Understand primary considerations when selecting a synthesis method and how these impact upon the conduct and reporting of a qualitative evidence synthesis/systematic review.

Programme What is Qualitative Systematic Review/ Evidence Synthesis MORNING What is Qualitative Systematic Review/ Evidence Synthesis What are your choices? Formulating a Question for Qualitative Review Searching & Sifting for Relevant Studies Break Quality Assessment [Plenary] Quality Assessment Workshop

Programme Introducing Thematic Synthesis/Framework Synthesis AFTERNOON Introducing Thematic Synthesis/Framework Synthesis Thematic Synthesis/Framework Synthesis Workshop Feedback Break Producing Your Review for Publication Where to Next? Close

What is Qualitative Systematic Review/Evidence Synthesis? Andrew Booth School of Health and Related Research, University of Sheffield, UK

Umbrella terms: Qualitative Systematic Review (but also non-quantitative, e.g. not meta-analysis) Qualitative Metasynthesis (Sandelowski et al, 2001) Qualitative Research Synthesis (Howell Major & Savin-Baden, 2010) Qualitative Evidence Synthesis (Cochrane Collaboration: Hannes & Lockwood, 2012)

61% from 2008-2012; 37.5% from 2010-2012 Topic=("qualitative systematic review" OR "qualitative evidence synthesis" OR "qualitative research synthesis") OR Topic=(metastudy OR metasynthesis OR "meta synthesis" OR "meta ethnography" OR "meta ethnographic" OR "metaethnography" OR "metaethnographic") OR Topic=("systematic review of qualitative") = 645 Currently about 10 per month

So what is a QES? qualitative evidence synthesis – an umbrella term increasingly used to describe a group of review types that attempt to synthesise and analyse findings from primary qualitative research studies (Booth et al, 2011).

Qualitative Evidence Synthesis Quantitative meta-synthesis, or meta-analysis, aims to pool numerical results of individual quantitative studies, qualitative meta-synthesis looks for “themes” or “constructs” that lie in or across individual qualitative studies. Within broader category of “qualitative meta-synthesis,” (Cochrane preferred term = qualitative evidence synthesis) narrow term “meta-ethnography” refers to specific method of data synthesis most widely adopted to date. Booth et al, 2006 9

Key Difference Goal not aggregative in sense of “adding studies together,” as with meta-analysis. Interpretative in broadening understanding of a particular phenomenon.

An Example Paterson and colleagues identified 38 studies examining first hand experience of living with diabetes. Prevailing metaphor = concept of “balance”. Specific sub-themes identified across multiple studies included “knowing one's body,” “learning how to manage diabetes,” and “fostering supportive, collaborative relationships with others.” 11

Stages of a Qualitative Evidence Synthesis? Formulating the review question Conducting a systematic literature search Screening and selecting appropriate research articles Analyzing and synthesizing qualitative findings Maintaining quality control Presenting findings (Sandelowski & Barroso, 2007) 12

Framework for systematic reviews of qualitative research (Garside, 2010) Stage Typical activities Developing research question Assemble team; Consult; Agree approach Scoping exercise Identify relevant research; Refine methods Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations Dissemination Target audiences; Limitations of review Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

Why do it? Perspectives of Patients, Relatives or Carers Of Condition Of Intervention Perspectives of Staff Implementation Barriers and Facilitators Patient Centred Outcomes Development and Testing of Theory Answering “What Works for Whom Under Which Circumstances?” Acceptability of Treatments Explaining Differential Effectiveness Designing and Evaluating Complex Interventions

Some Examples - 1 Perceptions of patients and physiotherapists on patient participation Systematic search in six databases using a set of key words, extracted relevant data, performed quality assessment and synthesized findings from selected studies. Narrative synthesis of  qualitative studies. Retained 11/160 studies. Two main themes: conceptualization of patient participation and patients' role preferences. Patient participation included goal setting, information exchange, decision-making and exercise training; often influenced power relation between patient and physiotherapist. Patients' willingness to participate varied; they often did not play their desired role. Schoeb & Bürge (2012)

Some Examples - 2 Reports possible incentives and barriers including: Illness symptoms, treatment effects, lack of support and negative staff attitudes as possible barriers; Symptom reduction, peer and staff support, knowledge, personal attributes and participation of staff as possible incentives. Incentives and barriers to lifestyle interventions for people with severe mental illness. Eight electronic databases [1985–March 2009] plus Cochrane Library and Google Scholar. Electronic ‘hand’ searches of key journals and explosion of references also undertaken. Narrative synthesis of qualitative, quantitative and mixed methods studies ROBERTS & BAILEY (2011)

Centres and Networks: EPPI-Centre Joanna Briggs Institute Cochrane Collaboration Campbell Collaboration Software EppiReviewer QARI Books Noblit & Hare (1988) - Meta-Ethnography Patterson et al (2001) - Meta-Study of Qualitative Health Research Sandelowski et al (2001) - Handbook for Synthesizing Qualitative Research Petticrew & Roberts (2003) - Systematic Reviews in Social Sciences Mays, Pope, Popay (2007) - Synthesizing Qualitative and Quantitative Health Research Howell Major & Savin-Baden (2010) - Introduction to Qualitative Research Synthesis Hannes & Lockwood (2011) - Synthesizing Qualitative Research: Choosing the Right Approach Saini & Schlonsky (2012) - Systematic Synthesis of Qualitative Research Booth et al (2011) – Systematic Approaches to a Successful Literature Review Gough, Oliver & Thomas (2012) - An Introduction to Systematic Reviews

School of Health and Related Research, University of Sheffield, UK What are your Choices? Andrew Booth School of Health and Related Research, University of Sheffield, UK

Specific Methods Bayesian Meta-Synthesis Critical Interpretive Synthesis Meta- Aggregation Meta Narrative Meta-Interpretation Matrix Method Meta Study Best Fit Synthesis Framework Synthesis Narrative Synthesis Realist Synthesis Meta-Ethnography EPPI-Centre Method Thematic Synthesis

Choice of Synthesis (Adapted from Noyes & Lewin, 2011) Decision to conduct a qualitative evidence synthesis To interpret synthesised qualitative evidence and develop explanatory theory or models Purpose of the additional qualitative synthesis or To aggregate/ summarise/ integrate qualitative data to address questions in relation to a specific intervention review Primarily to integrate and interpret qualitative and quantitative evidence within a single approach or integrated model. Can be used to develop explanatory theory. Framework synthesis Meta-ethnography Grounded theory Thematic analysis with theory generation Thematic analysis without theory generation Meta-aggregation Meta-summary Realist Review EPPI Approach Matrix Method Narrative Synthesis Bayesian Synthesis Critical Interpretive synthesis Best fit synthesis Product: Aggregated findings from source papers Product: Explanatory theory, analytical or conceptual framework or interpretative framework/mechanism

Bad Reasons for Choosing Method Frequency of Use of Method (e.g. Meta-Ethnography) Popularity/”Sexiness” of Method (e.g. Realist Synthesis) What a friend/ colleague/ mentor has used (once!) Bad experiences of others (may have been inappropriate!)

Key Considerations Role of Theory Available Expertise Extent of Description versus Interpretation Intended Product Other Considerations: Richness of Data and Available Time/ Resources

1. Will You Generate, Explore, Test Theory (Gough et al, 2012)? Generate – may require “suspension of disbelief” – quality assessment/ value judgement may come later (cp. Brainstorming)- Grounded Theory, Meta-ethnography Explore – looking for patterns - Narrative Synthesis, Thematic Synthesis Test – quality assessment differentiates well-supported and unsupported data - Framework Synthesis (incl. Best Fit Synthesis) NB. We (Carroll & Booth) are currently conducting empirical work on systematic identification of Theories

2. What Expertise Can You Access? Epistemological versus Technical: Expertise in Qualitative Research Methods (e.g. Grounded Theory; Framework Analysis, Thematic Analysis) Expertise in Synthesis Methods (incl. Searching, Data Extraction, Quality Assessment, Interpretation) Knowledge of Topic Area http://www.jiscmail.ac.uk/lists/asqus.html

3. Will You Describe or Interpret? All Reviews figure on a continuum between Description and Interpretation Description – What does the data say? – factual reporting of studies, themes etc… Reader does work of interpretation Interpretation – What does the data mean? –subjective interpretation of data and themes etc… Reviewer does work of interpretation – may be contested

4. What is Your Intended Output/Product? “the output of some methods of synthesis (Thematic Synthesis, textual Narrative Synthesis, Framework Synthesis, and ecological triangulation) is more directly relevant to policymakers and designers of interventions than the outputs of methods with a more constructivist orientation (Meta-Study, Meta-Narrative, Meta-Ethnography, Grounded Theory, CIS) which are generally more complex and conceptual” (Barnett-Page & Thomas, 2009) Thematic Synthesis (including Meta-Aggregation) and Framework Synthesis produce findings to directly inform practitioners (Thomas & Harden, 2009) Interpretive approaches (e.g. CIS, Meta-Ethnography) produce a model that requires practitioners to interpret relevance and applicability to their own context Narrative Synthesis or EPPI-Centre (matrix) methods may help to integrate and present quantitative/qualitative work

How Rich (“Thick”) is Your Data? Qualitative data from “thin” studies (or textual responses to surveys) will not sustain interpretive approaches Limited to Meta-Aggregation, Thematic Synthesis, Framework Synthesis, Narrative Synthesis –type approaches Rich/“Thick” reports will sustain Meta-Ethnography/Grounded Theory – may allow selective sampling/ theoretical saturation NB. Is “Unit of Analysis” Individual Study (Meta-Aggregation, Thematic Synthesis) or “Body of Evidence” (e.g. Meta-Narrative or Critical Interpretive Synthesis approaches) or even Theory (Framework Synthesis/Best Fit Synthesis)?

How Long?/How Much Have You Got? “Richer” approaches make fuller use of data – require fewer studies Meta-Aggregation, Thematic Synthesis can handle large numbers of studies Framework/best fit approach for speed 32 papers (775 patients and carers) reporting help-seeking experiences for at least 20 different types of cancer.

Synthesis Quiz For each of the following scenarios identify the review characteristics and try to match to an appropriate type of synthesis

Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product Scenario A: You are working in a general health technology assessment team conducting a synthesis of Group Therapy for Postnatal Depression. There are few studies and most reports are Nursing Standard/Times case reports. There is no prevalent theory. You will produce recommendations on what works for whom. Thematic synthesis Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product

Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product Scenario B: You are working as a group of topic experts and experienced qualitative researchers to examine the phenomenon of “Willingness to Hasten Death”. There are less than eight rich qualitative studies. No-one has yet conducted a synthesis to look at what is meant by the concept. Your review will help those who work in terminal care to gain a better understanding of the phenomenon. Meta-ethnography Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product

Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product Scenario C: You are a member of a government-funded Institute producing best practice reviews for nurses and nursing managers. You are asked to examine all evidence, quantitative and qualitative, for physical restraint in residential homes. You are expected to turn around a report in a very short time frame. Your final review is expected to include Recommendations for Practice. Meta-aggregation (or Thematic synthesis, but expectation is to do all studies) Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product

Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product Scenario D: Your review team, with access to clinical expertise, is commissioned to review the evidence on taking vitamins and supplements to prevent colorectal cancer. Because of the focus on long term outcomes there are few directly relevant qualitative studies. You decide to look for “related” studies and hypothesise attitudes to these specific agents from a wider evidence base. Attitudes to medicine are well-theorised but not within such a long-term prevention perspective. Your qualitative review will be a short chapter in a much longer effectiveness/cost effectiveness report. Best Fit or Framework Synthesis Role of Theory 2. Expertise 3. Describe/Interpret 4. Output/Product

Safest Options! If… There is a Pre-existing Theory or Framework…. Then ….Framework Synthesis (including Best Fit Synthesis) If… There is a Proximate (Close-ish!) Theory or Framework…. Then ….Best Fit Synthesis If… There is No Theory or Framework… Then …Thematic Synthesis (Can also act as first stage of Meta-Ethnography)

Formulating a Question for Qualitative Review Janet Harris School of Health and Related Research, University of Sheffield, UK

Framework for systematic reviews of qualitative research (Garside, 2010) Stage Typical activities Developing research question Assemble team; Consult; Agree approach Scoping exercise Identify relevant research; Refine methods Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations Dissemination Target audiences; Limitations of review Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

A 15 minute guide What is the focus? Brainstorm questions Refine Prioritize Determine next steps Reflect (Adapted from Rothstein and Santana (2011) Teaching Students to Ask Their Own Questions: One small change can yield big results. Harvard Educational Letter)

Focus and brainstorm Our topic of interest today is: Taking a client-centred approach to rehabilitation after stroke Our FOCUS – the problem that has initiated a review of qualitative research : Health professionals have different ideas about client-centred approaches to rehabilitation Our AIM: To produce evidence from a synthesis of the literature that will guide effective client centred-approaches BRAINSTORM Ask as many questions as you can about this topic Do not stop to discuss, judge, or answer the questions. Write down every question exactly as it is stated. Change any statement into a question

What did you come up with? Broad or narrow question? Open or closed? Advantages/disadvantages of the questions?

Prioritize Which questions do you want to pursue and why? Consider Likelihood of finding qualitative studies Relevance of studies to problem focus How the research synthesis will be used What I think I am looking for What people have researched

What I can realistically find!! Next steps would be… Conduct a ‘scoping search’ to see whether the published research seems to be a match for your problem focus Use the brief scoping of abstracts to refine the question further What I can realistically find!! What I think I am looking for

The final question for the qualitative review was: What are stroke survivors’ experiences with rehabilitation? (Peoples et al, 2011)

Searching and Sifting for Relevant Studies Andrew Booth School of Health and Related Research, University of Sheffield, UK

Framework for systematic reviews of qualitative research (Garside, 2010) Stage Typical activities Developing research question Assemble team; Consult; Agree approach Scoping exercise Identify relevant research; Refine methods Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations Dissemination Target audiences; Limitations of review Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

Relevant Studies… Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors), AND Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]

Other Considerations… How much literature is there? How is it characterised? e.g. Was it conceived as Qualitative Research? How far back will I go? Has the Intervention/Experience of Living with The Condition Changed? What Settings, Countries, Languages will I include? How important is Context?

http://www.nlm.nih.gov/nichsr/hedges/search.html Post acute stroke rehabilitation

Relevant Studies… Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors),

Matching the Focused Question Combination of natural language (e.g. stroke) and index terms (e.g. Cerebrovascular Disorders and Brain Infarction) May include subheadings e.g. /rehabilitation as well as index terms Need to optimise Sensitivity (Getting everything you want) and Specificity (Getting only what you want)

Sample Search Strategy (SIGN Guideline)

Relevant Studies… , AND Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]

Challenges Qualitative research is small part of the research literature Indexing in databases is inconsistent and variable Qualitative studies sometimes have creative titles or inadequate abstracts Medline might not be your preferred resource. If searching in limited databases try MEDLINE and CINAHL

How would you find qualitative studies?

Sources of Evidence 28/03/2017 © The University of Sheffield Bibliographic databases and catalogues Papers suggested by individuals Supplementary Methods Web and Grey literature searches 28/03/2017 © The University of Sheffield

Identifying Qualitative Research - Terminology Generic terms: e.g. “qualitative” plus “ESCAPADE” Exploratory Methods: Focus group, Grounded theory, Action Research, Content analysis, Thematic analysis Software: Nudist or NVivo Citations: Glaser & Strauss Application: Ethnology, Psychology Phenomenon: Perceptions, Attitudes, User Views, Standpoint, Viewpoint Approaches: Ethnographic Data: Stories, Narratives, Descriptions, Themes, Findings Experiences: Encounters, Experiences

What is a methodological filter? “A hedge or filter is a standardised search strategy that is designed to be used in conjunction with a subject search to retrieve valid studies from the (primary) medical literature”. Filters work in one of two ways: by identifying particular publication types or study designs most likely to answer a question by isolating subject or free-text terms most likely to be associated with high-quality studies

How do they work? Filters come from 3 different sources:- Subject heading Keyword Publication type

How do I use a methodological filter? Step One: Carry out a subject search as usual using subject headings and/or free text Step Two: Apply methodological filter appropriate to question you are asking One-line filter Maximum sensitivity filter Maximum specificity filter Mid-range filter

One-line filters MeSH Headings Keyword Publication Type e.g. Qualitative Research [Medline 2003-] Keyword “Findings” Publication Type Research [CINAHL only]

Methodological filters - 1 qualitative$ findings interview$ interviews.DE. 1 OR 2 OR 3 OR 4 Grant MJ. “How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research.” Health Info Libr J. 2004 Mar; 21(1):21-32.

Methodological filters - 2 Best sensitivity: exp interviews Best specificity: audiorecording.sh. Best optimization of sensitivity and specificity: exp study design Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res. 2007 May;17(5):705-10.

Methodological filters - 3 Don’t forget MeSH heading Qualitative Research Introduced in 2003, so coverage is limited But you can combine it with one of the previous filters by using the “OR” operator

Sources of filters ISSG search filters resource http://www.york.ac.uk/inst/crd/intertasc/ SIGN http://www.sign.ac.uk/methodology/filters.html McMaster Hedges Project http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx PubMed Clinical Queries http://www.ncbi.nlm.nih.gov/corehtml/query/static/clinical.shtml PubMed Clinical Queries screen enables searching using stored filters - Therapy, Diagnosis, Etiology and Prognosis McMaster University Ontario, Canada, Health Information Research Unit, Evidence Based Health Informatics.

Each Question requires a different solution... Findings showed that a simple search strategy (broad-based terms - 3 search terms) was as effective as a complex one (free text - 48 search terms) in locating qualitative research on patients’ experiences of living with a leg ulcer. May be feasible to restrict searches with a clear nursing focus to CINAHL bibliographic database. Replication with other nursing topics is required. Flemming K, Briggs M. Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs. 2007 Jan;57(1):95-100.

“Supplementary” Methods Citation searches Reference lists Hand searching 28/03/2017 © The University of Sheffield

Citation searching- Google Scholar

In Conclusion Cp. Cluster searching! Identifying Qualitative Research is Challenging Simple strategy with qualitative terms (e.g. Interview*; qualitative; findings etc) may be more effective than exhaustive lists of terms May not be necessary to search as many databases as quantitative topics But supplementary methods (e.g for books and theses) may be very important! Cp. Cluster searching!

School of Health and Related Research, University of Sheffield, UK Reading the Reports Janet Harris School of Health and Related Research, University of Sheffield, UK

Aims and objectives What do we mean by quality assessment? What does it involve? How do you do it? What are the strengths and weaknesses of different tools or checklists?

Framework for systematic reviews of qualitative research (Garside, 2010) Stage Typical activities Developing research question Assemble team; Consult; Agree approach Scoping exercise Identify relevant research; Refine methods Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations Dissemination Target audiences; Limitations of review Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

Quality assessment: Basic criteria Carrying out ethical research Importance of the research Clarity and coherence of the report Use of appropriate and rigorous methods Importance of reflexivity or attending to researcher bias Importance of establishing validity or credibility Importance of verification or reliability Cohen focuses on several key criteria – quality of reporting what was done, the researcher’s stance in relation to the research, validity and reliability (Cohen et al, 2008)

Quality assessment: 3 stages Filtering against minimum criteria, involving adequacy of reporting detail Assessing technical quality of the study’s elements - methodological soundness, credibility, validity Judging theoretical consistency Quality assessment needs to follow three stages. The first focuses on reviewing whether the conduct of the study is reported in enough detail to judge rigour. The minimum criteria for what needs to be reported are set out in assessment checklists such as CASP or JBI. As mentioned earlier, these may be modified by authors to include additional criteria for what needs to be reported. If reporting is adequate, then authors can judge the quality of how the study was conducted. As a last stage, the consistency with which theory was developed can be judged.

Filtering: Relevance and adequacy of reporting Relevance: generally judged against title and abstract in the first instance Reporting: Is the study process reported well enough to judge technical rigour? This depends on your area of interest… Intervention Core elements of the intervention as planned Non-core elements e.g. components that are added to suit a particular context or/or population Theory development: Relationship between Context, Mechanisms and Outcomes Implementation: barriers and facilitators to implementing the intervention First, if the information is not reported, then it is of little use in terms of informing an effectiveness review. This may seem obvious, but the results of a search should be used to scope the completeness of information on the area of interest. The core components of a complex intervention, for example, may be incompletely reported, making it difficult to judge reasons for effectiveness. Similarly, studies may not have reported context, which could have an influence on outcomes. Where reporting is adequate, you can go on to judge the technical rigour of the research

Quality appraisal: Basic criteria Technical quality – Qualitative concepts Techniques Credibility: the representation of data fits the views of the participants studied, the findings hold true outside auditors or participants validate findings (member checks) peer debriefing, attention to negative cases, independent analysis of data by more than one researcher verbatim quotes persistent observation (stay in the field long enough) Confirmability: findings are qualitatively confirmable through the analysis being grounded in the data, through examination of the audit trail assessing the potential effects/impact of the researcher during all steps of the research process Reflexivity toward personal influences, bias providing background information on the researcher’s background, education, perspective, school of thought Dependability: process of research is logical, traceable and clearly documented, particularly on the methods chosen and the decisions made by the researchers peer review, debriefing, audit trails triangulation, the use of different methodological approaches to look at the topic of research reflexivity to keep a self-critical account of the research process calculation of inter-rater agreements Transferability: research findings are transferable to other specific settings providing details of the study participants to enable readers to evaluate for which target groups the findings potentially hold true providing contextual background information, demographics providing thick description about both the sending and the receiving context Quality appraisal: Basic criteria Credibility evaluates whether or not the representation of data fits the views of the participants studied, whether the findings hold true. Confirmability evaluates the extent to which findings are qualitatively confirmable through the analysis being grounded in the data and through examination of the audit trail. Dependability evaluates whether the process of research is logical, traceable and clearly documented, particularly on the methods chosen and the decisions made by the researchers. Transferability evaluates whether research findings are transferable to other specific settings. IN other words, these are criteria for assessing internal and external validity in qualitative research

Validity: linking data collection and analysis to conclusions When evaluating methodological soundness, we need to know whether the set of arguments or the conclusion derived from a study necessarily follows from the premises. whether it is well grounded in logic or truth. whether it accurately reflects the concepts, the ideas that it is intended to measure. Technical rigour is about validity – whether there is a clear pathway in terms of translating data into tudy findings that lead to a logical conclusion. Statements need to be grounded in the evidence and we definitely need to know whether statements from a researchers accurately reflect the ideas of the participants that it is intended to reveal.

Theoretical quality How was the theory developed? Does your review question aim to develop an explanation for something – a theory? If it does, then you need to consider theoretical quality How was the theory developed? Is there congruence between the data and the interpretation of findings? Can you see a logical connection or translation of findings into theory?

Revisiting relevance? Relevance can be a multistage process The first stage involves excluding obviously irrelevant papers (filtering) - The second stage involves weighing relevance to the review question with the quality of reporting and technical competence at establishing validity Data may be relevant and insightful but not rigorously analyzed...do you include it?

QARI Critical Appraisal Instrument Criteria 1) There is congruity between the stated philosophical perspective and the research methodology. 2) There is congruity between the research methodology and the research question or objectives. 3) There is congruity between the research methodology and the methods used to collect data. 4) There is congruity between the research methodology and the representation and analysis of data. 5) There is congruity between the research methodology and the interpretation of results. 6) There is a statement locating the researcher culturally or theoretically. 7) The influence of the researcher on the research, and vice-versa, is addressed. 8) Participants, and their voices, are adequately represented. 9) The research is ethical according to current criteria or, for recent studies, there is evidence of ethical approval by an appropriate body. 10) Conclusions drawn in the research report do appear to flow from the analysis, or interpretation, of the data. TOTAL We are going to use the criteria developed by the Joanna Briggs Institute Quality Assessment and Review Instrument (QARI). You will each get a copy of this checklist. As you read the article, decide whether each criteria is met using Yes No or Unclear. Note your reasons for making these decisions and count up your score. We are using a paper that was considered for the qualitative review by Peoples et al (2011) Stroke survivors’ experiences of rehabilitation The paper is: Proot et al (2000) Patient autonomy during rehabilitation: the experiences of stroke patients in nursing homes. International Journal of Nursing Studies 37:267-276

Criteria categorization and definition Study design: Yes - if e.g., “a case study approach was used . . .”, “phenomenology was used . . .” No - If paper does not specify study design Unclear if unsure Method of data collection: Yes - If details of the data collection method are given e.g., piloting; interviews, topic guides for interviews; number of items in a survey; use of open or closed items; validation, and so forth. No - If only states “focus group”, “interviews were used” or “questionnaire was used” Unclear if unsure  Selection of participants: Yes - If the sampling and recruitment of participants is described in full or explicitly as e.g., purposive, convenience, theoretical and so forth. No - If only details of participants are given, e.g. age, gender, number Methods of data analysis: Yes - If full details of analysis method are given, e.g., transcription and form of analysis (with reference or full description of method), validation tests, and so forth. No - If only states “content analysis” or that “data were analyzed” Quality of Reporting Tool (Carroll et al, QHR 2012)

Appraising a paper for a qualitative review Group 1: Use the QARI appraisal tool Group 2: Use the Quality of Reporting Tool Make your decision about the quality of the paper Technical quality Relevance Theoretical quality

Integrating papers Would you use this paper in the qualitative review? Poor methodological quality but relevant data and valuable insights Good quality, but poor interpretation and limited insight Which gets privileged – the quality score or useful insight?

Learning outcomes An understanding of What we mean by quality assessment What critical appraisal involves How you conduct an appraisal The strengths and weaknesses of different tools or checklists Which tool might be best for you

Introducing framework synthesis and thematic synthesis Chris Carroll School of Health and Related Research, University of Sheffield, UK

Aims and objectives What do we mean by framework synthesis and thematic synthesis? How do you extract data for synthesis? How do you perform both types of synthesis? How do you “complete” the synthesis?

Acceptability and appropriateness: may explain compliance Time Content 13.15-13.35 Introduction to two types of synthesis 13.35-14.45 Extracting and coding data: Introduction Practical (10 mins) Feedback (5 mins) How to do thematic analysis Practical (15 mins) Feedback (10 mins) Analysis and synthesis 14.45-15.00 Feedback 15.20-15.45 Producing your review for publication (AB) Acceptability and appropriateness: may explain compliance

Framework for systematic reviews of qualitative research (Garside, 2010) Stage Typical activities Developing research question Assemble team; Consult; Agree approach Scoping exercise Identify relevant research; Refine methods Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations Dissemination Target audiences; Limitations of review Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

A Framework

How do you create a framework? Develop own framework or themes or concepts for coding Oliver S et al: A multidimensional conceptual framework for analysing public involvement in health services research. Health Expectations 2008, 11:72-84. Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006. Identify or create a thematic framework or conceptual model Carroll C, Booth A, Cooper K. A worked example of “best-fit” framework synthesis: A systematic review of views concerning the taking of potential chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29

An existing conceptual framework Conrad K, et al (1996). The worksite environment as a cue to smoking reduction. Research in Nursing & Health, 19 21-31.

A Framework

Thematic synthesis Framework synthesis

The case study The question: What are the barriers and facilitators of patient autonomy post stroke? The studies: Proot I et al, Patient autonomy during rehabilitation: the experiences of stroke patients in nursing homes, International Journal of Nursing Studies, 2000; 37: 267-76. Mangset M et al, ‘We’re just sick people, nothing else’: ... Factors contributing to elderly stroke patients satisfaction with rehabilitation, Clinical Rehabilitation, 2008; 22: 825-35. Studies identified using usual methods – as covered this morning

Data extraction What is it? An attempt to reduce a mass of material (your included papers) to a much smaller body of text and numbers, amenable to analysis and the interpretation of findings Why extract these data? Same as for any review? Combining? Sensitivity analysis?

Data extraction form Location Setting Sample (n) Age, Gender, Ethnicity and other relevant variables (e.g. Marital status, socio-economic status) Intervention (if any) Outcomes and results Study design Data collection method Quality assessment criteria Further citations Format????? Why extract these data? Same as for any review? Combining? Sensitivity analysis? Contextualising and interpreting your findings

Example form: Proot 2000 Why extract these data? Same as for any review? Combining? Sensitivity analysis?

What results do you extract? What is your question? Question: What are the barriers and facilitators of patient autonomy post-stroke? Keep the question in mind as you read: Are the data relevant to this question? Is the question answered by the data?

Extraction of results Results sections of included studies: Authors’ statements clearly-supported by data and/or Verbatim quotations “... It was often difficult to distinguish [Schutz’s] first- from second-order constructs” Atkins et al. Conducting a meta-ethnography of qualitative literature: Lessons learnt, BMC Med Res Methodol 2008; 8: 21 Schutz A. Collected Papers 1971; vol.1: 361. 1st order construct: participants; 2nd=authors, but because of selectivity by authors, arguably all 2nd order

1st order construct: participants; 2nd=authors, but because of selectivity by authors, arguably all 2nd order – Morse??? 3rd=reviewers From: Mangset p.829

Extracting data for thematic analysis

Extracting data for thematic analysis

Extracting or “coding” data using framework analysis Framework developed from ... Kirkevold M et al. Promoting psychosocial well-being following a stroke: developing a theoretically and empirically sound complex intervention, International Journal of Nursing Studies, 2012; 49: 386-97.

Extracting or “coding” data using framework analysis Code number Theme Definition 1 Dignity A self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself 2 Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness 3 Coping Ability to adapt and cope within home and hospital settings 4 Family Support Relations with immediate family 5 Support for patient’s coping efforts Patients take responsibility and are supported by others in this 6 Bodily changes and impairments Difficulties generated by residual or post-stroke conditions Handout 5

Extracting or “coding” data using framework analysis

Extracting or “coding” data using framework analysis

Extracting or “coding” data using framework analysis

Extracting or “coding” data using framework analysis

Extraction and coding: Practical Consider the full framework in Handout 5 Consider extracts A, B and C on Handout 6 Code the extracted data according to the framework by entering the number of the framework code in the right-hand column on Handout 6 Note: Data may be assigned more than 1 theme No code in the framework, in your opinion, might capture, “fit” or reflect the data. If this is the case, enter “New theme” into the right-hand column 10 minutes: Individual and compare with neighbour; 5 minutes: Discussion of results and experiences Does it make sense? Was it easy? Was there consistency? If so, what are the implications of this for extraction and synthesis?

What now? Secondary thematic analysis Practical: Thematic analysis Discussion of practical Synthesising your data

What now? You have extracted your data AND You have coded your data against your framework (framework analysis) Or you may have coded some of your data but not others (“best fit” framework synthesis) OR You are conducting thematic synthesis anyway ... You now need to assign themes to those data without themes, i.e. Using secondary thematic analysis (data reduction, display, conclusion) How do you do it? 3 stages from Miles and Huberman 1994

Generating new themes “Evidence does not speak for itself. It requires interpretation .... on the basis of our personal experience and a range of ideas that we use to make sense of our observations” Kelly M et al. Evidence-based public health: A review of the experience of NICE developing public health guidance in England, Social Science and Medicine, 2010; 71: 1056-62.

Developing new themes Other / New theme? Interpret Create Revisit and revise Discuss Agree ... = Learning self-help skills / guided self-determination C. “…“I will take that paper when P. is finished with it. I want to study it. Then I'll take my shoes off, and practise as I see it in the picture” (p04b, p. 11), Proot, p.271

Thematic analysis: Practical Consider extracts D, E and F on Handout 6 Develop and assign new themes to these extracts and enter them in the right-hand column on Handout 6 Note: Data may be assigned more than 1 theme There may also be code in the framework which, in your opinion, might capture, “fit” or reflect the data. If this is the case, also enter this code into the right-hand column 15 minutes: Individual and compare with neighbour; 10 minutes: Discussion of results and experiences Does it make sense? Was it easy? Was there consistency? If so, what are the implications of this for extraction and synthesis?

Generating new themes: Practical Feedback Interpretation Bias? Reviewers might have agendas too What is the result of the secondary thematic analysis? Alot written on different theoretical perspectives of primary researchers, but applies equally to reviewers interpreting data

Synthesis Report the results of the new thematic framework Narrative structured by theme Report each theme with reference to studies and data in order to: Specify the evidence base for the theme Illustrate the theme with reference to actual data McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):130-153

Synthesis

Synthesis Reduce the findings into a smaller number of categories and then a single finding (metasynthesis, according to the Joanna Briggs Institute approach): McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):130-153

Synthesis The result might be a new theory

A new framework Code number Theme Definition 1 Dignity 2 Well-being 3 A self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself 2 Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness 3 Coping Ability to adapt and cope within home and hospital settings 4 Family Support Relations with immediate family 5 Support for patient’s coping efforts Patients take responsibility and are supported by others in this 6 Bodily changes and impairments Difficulties generated by residual or post-stroke conditions 7 Self-determination Define theme as a concept, based on data 8 Need for information 9 Lack of shared decision-making 10 Acceptance of therapy Cluster

A new framework and synthesis Theme Categories Dignity Psychological factors Well-being Self-determination Coping Bodily changes and impairments Physical factors Need for information Interaction with professionals and family Lack of shared decision-making Family Support Support for patient’s coping efforts Acceptance of therapy Cluster, but very linear

Synthesis Alternatively, go beyond the framework to create a new conceptual model or theory ... Revisit data to explore the relationships between the themes or findings of your framework The “richer” or “thicker” the data, the deeper the resulting model or theory Narrative and diagrammatic representations Not reporting a gap analysis here … principal differences are:

Synthesis The result might be a new theory

Do you stop there? Test your synthesis? Disconfirming cases Booth A, Carroll C, Illott I. Desperately Seeking Dissonance: Identifying the “Disconfirming Case”, Qualitative Evidence Synthesis, Qualitative Health Research 2012 (in press). Morse, J. M. The significance of saturation. Qualitative Health Research 1995; 5, 147-149. Sensitivity analysis By quality, population, location, setting etc. By frequency and “thickness” Carroll C, Booth A, Lloyd-Jones M. Should we Exclude Inadequately-reported Studies from Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews, Qualitative Health Research 2012; 22(10) Boeije et al (2011) Making a difference: towards a method for weighing the evidence in a qualitative synthesis, Journal of Evaluation in Clinical Practice 2011 17(4):657-63 Sexual health services: in the same study, we had students who used the service because they trusted its confidentiality and students who did not use the service because they did not trust its confidentiality

What is the purpose of your review and synthesis? To answer questions and to solve problems that cannot be addressed by quantitative research To illuminate patient and client experience and to develop theory To be useful

Learning outcomes An understanding of What we mean by framework synthesis What we mean by thematic analysis How you conduct framework and thematic synthesis, or “best-fit” framework synthesis The strengths and weaknesses of the different methods How you “complete” your synthesis

Summary Extract and interpret data Synthesis Test your synthesis Integrating and writing-up ...

Producing Your Review for Publication Andrew Booth School of Health and Related Research, University of Sheffield, UK

Framework for systematic reviews of qualitative research (Garside, 2010) Stage Typical activities Developing research question Assemble team; Consult; Agree approach Scoping exercise Identify relevant research; Refine methods Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations Dissemination Target audiences; Limitations of review Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

Between Two Traditions Systematic Review Publishing Guidelines (e.g. PRISMA – formerly QUOROM) Primary Qualitative Research Reporting Traditions Systematic Reviews QES Qualitative Research

Requirements for a Published QES Systematic Review requires: Explicit methods Transparency Audit Trail Review Question Search strategies & sources Quality Assessment Method of Synthesis Strategies to reduce bias Primary Qualitative Research requires: “Believability” Findings Grounded in the Data Themes/Constructs Selective Findings Reflexivity

Methods of Appraisal/ Extraction Purpose of Study Methods of Synthesis Search Strategy Databases and Keywords

PRISMA Flow Diagram

NB. This is non-standard but informed by QUOROM/ PRISMA

Themes Findings Studies

Narrative synthesis Verbatim Extract Unsubstantiated Comment

Relationship between Themes

Structuring Recommendations

School of Health and Related Research, University of Sheffield, UK Where to Next? Andrew Booth School of Health and Related Research, University of Sheffield, UK

Further Reading Howell Major & Savin-Baden (2010) Hannes & Lockwood (2011)

Further Resources http://www.mendeley.com/groups/518691/cochrane-qes-register/

Further Support http://www.jiscmail.ac.uk/asqus http://cqrmg.cochrane.org/

Further Training Look out for SyNTAQuES – II (the Intermediate Version) from the Higher Education Academy http://www.heacademy.ac.uk/events Advanced Training is available from: ESQUIRE13, September 2nd-4th 2013, ScHARR, University of Sheffield http://www.sheffield.ac.uk/scharr/shortcourseunit Previous materials at: http://esquiresheffield.pbworks.com

Some Issues to Think About! Or it’s not all that simple really!

Qualitative Systematic Review Qualitative Data Explicit? (cp. Iterative) Comprehensive? Qualitative Research? Reproducible? (cp. Reflexivity/ Subjectivity/ Interpretation) (cp. Theoretical Saturation; Purposive Sampling) Particular Type of Study (e.g. all Grounded Theory Studies)? Follows “System”: from Systematic Review or from Primary Qualitative Research? With/Without Quantitative Research? Qualitative Data?