1. Systematic Literature Reviews: A taxonomy of designs and procedures
Miriam Bender, PhD RN
Assistant Professor
Sue & Bill Gross School of Nursing
University of California Irvine
3/14/2017
Nursing Science Scholar Forum

2. Agenda Literature review: What and Why
Types of systematic literature reviews
Steps of the systematic literature review
Examples along the way
Q&A (and any time you have a question!)

3. What is a systematic review?
“A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyse data from the studies that are included in the review” (CRD, 2001) Product is an exacting synthesis on a topic of interest
Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

4. Systematic vs. “Lit” reviews
Scientific approach to a review article
Criteria determined at outset
Comprehensive search for relevant articles
Explicit methods of appraisal and synthesis
Meta-analysis may be used to combine data
Depend on authors’ inclination (bias)
Author gets to pick any criteria
Search any databases
Methods not usually specified
Vote count or narrative summary
Can’t replicate review

5. Why are they important?
“A substantive, thorough, sophisticated literature review is a precondition for doing substantive, thorough, sophisticated research”
They contribute to a body of knowledge
Levels of evidence
Conceptual frameworks
Identify gaps
Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational researcher, 34(6), 3-15.

6. Why are they important? Delimiting the research problem
Establishing the context of the topic or problem
Seeking new lines of inquiry
Avoiding fruitless approaches
Gaining methodological insights
Identifying recommendations for further research
Discovering important variables relevant to the topic
Synthesizing and gaining a new perspective

7. Why do a systematic review?
PhD: It is good scientific practice to articulate how YOUR study will contribute to the body of knowledge
DNP: it is good evidence based practice to understand HOW the evidence will ‘play out’ in your setting

8. Why do a systematic review?
There is no better project for developing a scholarly mind
“SCHOLARS BEFORE RESEARCHERS”
Get practice in detailed, comprehensive reading, sorting, writing
Learn NOT TO SKIP STEPS; there are no shortcuts to good scholarship
Understand theory/concepts and why important
Knowledge generation as incremental and iterative

9. Caveat
“The dirty secret known by those who sit on dissertation committees is that most literature reviews are poorly conceptualized and written” (Boote and Beile, 2005, p. 4).
Why?
They are hard to do
They take experience, but students are novices
It can seem very overwhelming at first
Expect 3-6 months concerted activity
Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational researcher, 34(6), 3-15.

10. Who should be involved? A team is best
Stakeholders: what is important to know?
Experts: researchers, lit reviewers, etc.
Librarian
Might have to be ‘only author’ (i.e. PhD students), but if not, consider many players to collaboratively conduct the work

11. Types of Systematic Reviews
A review that aims to comprehensively identify all relevant studies to answer a particular question, and assesses the validity (or ‘‘soundness’’) of each study taking this into account when reaching conclusions
Narrative Analysis
Sometimes used to refer to a systematic review that synthesizes the individual studies narratively (rather than by means of a meta-analysis). This involves systematically extracting, checking,
and narratively summarizing information on their methods and results
Conceptual review
A review that aims to synthesize areas of conceptual knowledge that can contribute to a better understanding of these issues. The objectives of these syntheses are ‘‘To provide an overview of the literature in a given field, including the main ideas, models and debates
Rapid review
A literature review carried out (often systematically) but within a limited time (sometimes weeks or a couple of months) and with restrictions on the scope of the search (for example, restricted by year, or country)
Realist review
Approach to reviewing studies which involves synthesizing individual studies with a view to producing generalizable theories (rather than synthesizing outcomes across studies (as systematic reviews do)
Scoping review
A review sometimes carried out in advance of a full systematic review to scope the existing literature – that is to assess the types of studies carried out to date, and where they are located. This can help with refining the question for the full review, and with estimating the resources that will be needed.
Critical review
Term sometimes used to describe a literature review that assesses a theory or hypothesis by critically examining the methods and results of the primary studies, often with a wealth of background and contextual material, though not using the formalized approach of a systematic review

12. Steps of a Systematic Review
Define the research question
Determine the types of relevant literature
Search for the literature
Screen the findings: inclusion/exclusion
Extract the data
Critically appraise the articles
Synthesize the data extracted
Write up the results

13. 1. Define the research question
Effectiveness:
Does the intervention work/not work?
Who does it work/not work for?
Other important questions:
Determinants? Theories? Concepts? Variables?
Takes context into account
What works or doesn’t, and where?
Takes complexity into account
Macro-micro levels of influence
Quantitative and qualitative data

14. 1. Define the research question
PICO great tool!!
Population
Individuals? Community?
Intervention
Simple? Complex?
Comparison
Other interventions? Null hypothesis?
Outcomes
Proximal, Distal
Other Questions
How does X work?
What are facilitators/barriers to implementation of X?
Is X cost effective
Is X appropriate for a specific population/content?
How have constructs been measured?

15. 2. Determine the types of relevant literature
Effectiveness questions
Hierarchy of evidence: what levels will you include?
Process questions
Qualitative, descriptive, mixed methods research
Etiology questions
Observational research

16. 2. Determine the types of relevant literature
Its about the BEST EVIDENCE, not necessarily only a ‘gold standard’
Quantitative
Qualitative/mixed methods
Does X work? +
Does X work better than Y?
How does X work?
What are facilitators/barriers to implementation of X
Is X safe?
Is X cost effective?
Is X appropriate for a specific population/context?
Do people like X?

17. 3. Search the literature Develop a search strategy
Always work with your librarian!!
PICOC terms, free text, Mesh terms, etc.
Boolean terms
Specific study designs as filters? Or the phenomenon as a filter? Dates, language
Databases of relevance?
Pubmed, CINAHL, center for reviews and dissemination, Cochrane, Joanna Briggs, Campbell Collaboration, Google Scholar
Will need a search strategy for each database
The gray literature?
Reference lists, Google scholar, professional organization white papers, conference proceedings, dissertations

18. 3. Search the literature
Need to balance sensitivity (recall) with specificity (precision)
Scoping review: Piloting the lit review
What kinds of studies are out there?
i.e. what actually is relevant?
What kinds of populations/outcomes researched?
Surrogate outcomes?
Variability in PICOC items?
Different theories, concepts?
Position papers?

19. 3. Search the literature Documentation is critical
The search terms from each database
Names of the databases searched
The date when the search was run
Filters sued
How many hits

20. TIPS & TRICKS Import each search separately into a reference app
Zotero, Mendelay, Papers (NOT endnote)
It takes the different formats of each database and converts it to a standard metadata format
Then import into an excel file to work with
Combine searches, remove duplicates

21. 4. Screen the findings: inclusion/exclusion
Read the titles
Are any patently not related to your research question?
Read the abstracts
Right PICO elements? Right ‘other’ elements?
New elements not
Don’t exclude yet unless obvious to exclude
Read the papers
Will probably be extracting data at this point
2 birds with one stone, make sure you don’t miss anything
Type of paper, Aims, methods, population, concepts/variables

22. 4. Screen the findings: inclusion/exclusion
Document yes/no for each paper, and WHY
Will create short labels in your excel file
This is what goes into the consort diagram

23. 5. Extract the data Extracting details in standardized way
Based on Research Question
Sample, setting
Methods, instruments, procedures
Data collected, analyses completed
Theories, concepts, frameworks
Outcomes: how operationalized and measured
Levels of analysis
More “columns” of data as get more knowledgeable about your literature
A framework/algorithm might help
Implementation science models
Social ecology framework

24. 5. Extract the data Study characteristics Participant characteristics
Aims, design, inclusion criteria, recruitment methods, unit of analysis,
Participant characteristics
Age, gender, ethnicity, comorbidities
Interventions and setting
Dose, reach, duration
Process and Outcome measures
Instruments, data, procedures, outcomes
Ways operationalized constructs
Theories/concepts used
Implementation factors
Levels of analysis

25. Examples

26. Example
Assessing implementation in a meta analysis of effectiveness of youth mentoring programs
Data extracted much more than methodological
Monitoring of program implementation
Matching mentors of young people
Ongoing training, supervision and support of mentors
Parental involvement
Frequency of contact
Length of relationship

27. 6. Critically appraise the articles
Is the article worth including into your review?
Methodology ok?
Appropriate focus?
Valuable information, regardless of study quality?
Flawed research versus relevant information
Implementation data?
Use validated quality appraisal tools
Will be different based on type of study
For example:

28. 6. Critically appraise the articles
Bias critical if determining efficacy
Biased studies will ‘muddy’ the findings
May continue to ‘exclude’ articles at this stage
Not so critical for other questions
Theories used, concepts, variables, implementation factors, etc.
Description might be critical, even if study overall is biased
Do higher quality studies operationalize important elements differently than lower quality studies?
Each study will get some sort of “score”
Use this score when deciding how to include and/or synthesize studies and data across studies

29. Example

30. Might take the longest
Data extraction and critical appraisal will happen iteratively
Will take time to find the relevant data
Inclusion/exclusion may take some time
Data extraction/quality appraisal are analytic procedures
Will be learning about your topic
Identifying what is important
Generating hypotheses, insights, ideas
Finding the pearls through all the sifting

31. 7. Synthesize the data extracted
Quantitative and/or qualitative/narrative analysis
Can do both
Start with the data extraction spreadsheet
May be transformed into many different tables
Descriptive table of included articles (i.e. table 1)
Critical appraisal
CAUTION: TABLES ARE NOT THE SYNTHESIS
They are the tools used to write up the synthesis
The tables are the starting point, not the endpoint of your synthesis

32. Tips & tricks Learn excel and use it! Keep everything in 1 file
Sorting, hiding, formulas for finding duplicates
Keep everything in 1 file
Never delete tabs: copy them and work on the copy
Chain of record, can audit, backup
Each table can be a different tab
Same extraction data analyzed different ways

33. Quantitative Analyses
Are the studies similar enough to conduct a meta analysis?
Can a comparison be made across studies?
Variability in populations, interventions, settings, outcomes, study designs
Differential effectiveness??
Equity review (Place, race, occupation, religion, education, socioeconomic status, social capital)
Moderator effects
Subgroup analysis (caveat, unreliable many times)
Meta regression

34. Quantitative Analyses
Quantitative analysis
Overall Effect size: quantitative meta analysis
frequency distribution, funnel plots, forest plots, etc.
Schematic analysis
Range, quality and type of research evidence available
Gap analysis
Where is the data missing?

35. Approaches tabulation
A common approach, used to represent data visually. The way in which data are tabulated may affect readers’ impressions of the relationships between studies, emphasizing the importance of a narrative interpretation to supplement the tabulated data.
Moderator variables and subgroup analysis
This refers to the analysis of variables which can be expected to moderate the main effects being examined in the review. This can be done at the study level, by examining characteristics that vary between studies (such as study quality, study design or study setting) or by analyzing characteristics of the sample (such as subgroups of participants).
Groupings and clusters
The included studies might be grouped at an early stage of the review, though it may be necessary to refine these initial groups as the synthesis develops. This can also be a useful way of aiding the process of description and analysis and looking for patterns within and across groups. It is important to use the review question(s) to inform decisions about how to group the included studies.
Transforming data into common measure
In both narrative and quantitative synthesis it is important to ensure that data are presented in a common measure to allow an accurate description of the range of effects.

36. Example

37. Narrative Analysis
Textual approach that provides an analysis of the relationships within and between studies and an overall assessment of the robustness of the evidence
Integrates different types of evidence to produce meaningful synthesis of the findings
Steps:
Organize studies into logical categories
Analyze findings within each category
Synthesize findings across all studies

38. Approaches Find a framework to help organize data
Change theory
Implementation theory
Complexity theory
Complex healthcare intervention
Conversely: end product becomes a conceptual framework
Identify what concepts are important

39. Approaches Concept mapping Qualitative case description
Conceptual triangulation
Thematic analysis
Meta ethnography
Grounded theory
Qualitative meta synthesis
Qualitative comparative analysis
All are very sophisticated analyses that involve translating data into themes, concepts, patterns, theories, models, etc.
This could be a dissertation study in and of itself

40. 8. Write up the results Prioritize findings
Might have more than 1 paper
i.e. outcomes review, implementation review
Use Equator Network for reporting
To include
Strengths and weaknesses of review
Meaning of the findings; what do we know and what still remains to be examined?
Implications: more research? Of what kind?

41. “Facts rarely speak for themselves”
Writing up a lit review will be one of the biggest scholarly challenges you initially face
“in the trees”
Overwhelmed with everything needed to write
Remember, tables are not the synthesis
Unsure how to organize the manuscript
What goes in the intro, and what in the discussion?

42. Writing the analysis
Establish to what extent existing research has progressed towards clarifying a particular problem
Identify relations, contradictions, gaps, and inconsistencies in the literature, and explore reasons for these (e.g. by proposing a new conceptualization or theory which accounts for the inconsistency)
Formulate general statements or an overarching conceptualization
Make your own point, rather than summarizing all the points everyone else has made
Comment on, evaluate, extend, or develop theory
provide implications for practice and policy
Describe directions for future research
i.e. your dissertation research
i.e. the format of your DNP project

43. Tips and tricks
Find a systematic review you like and use as a template
Write your paper AS you do the review
Purpose statements
Research aims
Methods
Inclusion criteria, search strategy, screening
Get the consort diagram done early!
Results
Get Table 1 done and described early in the game
TAKE NOTES during analysis: these become sentences in the review
Implications
Write the need for your specific research study!

44. Summary
The systematic literature review is your entrée into the world of doctoral scholarship
Teaches you detail orientation, patience, the pitfalls of ‘cheap’ research, rigor, exactitude
It will reap rewards for years to come
PUBLISHABLE!
Can easily summarize for grants
Will be the TOP expert in your specific field afterwards
Nobody on the planet will have done the work you did to answer your research question!
Will have ALREADY conducted research before you ‘get started’ on your research/project

45. Resources/references
Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. Blackwell Publishing Ltd.
Center for reviews and dissemination. (2009). Systematic reviews: CRD's guidance for undertaking reviews in health care. CRD, University of York.
Cochrane. EPOC specific reources for review authors. Retrieved from
Thomas, B. H., Ciliska, D., & Dobbins, M. (2016). A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. … On Evidence‐Based …, 1–9.
Randolph, J. (2009). A guide to writing the dissertation literature review. Practical Assessment, Research & Evaluation, 14(13).
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., ... & Duffy, S. (2006). Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version, 1, b92.
Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services Research & Policy, 10 Suppl 1, 6–20.
Bravata, D. M., McDonald, K. M., Shojania, K. G., Sundaram, V., & Owens, D. K. (2005). Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Annals of Internal Medicine, 142(12 Pt 2), 1056–1065.
Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Medical Research Methodology, 12(1),
Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., et al. (2006). Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Medical Research Methodology, 6(1), 35.
Thorne, S. (2009). The role of qualitative research within an evidence-based context: Can metasynthesis be the answer? International Journal of Nursing Studies, 46(4), 569–575.
Goldsmith, M. R., Bankhead, C. R., & Austoker, J. (2007). Synthesising quantitative and qualitative research in evidence-based patient information. J Epidemiol Community Health, 61(3), 262–270.
Entwistle, V., Firnigl, D., Ryan, M., Francis, J., & Kinghorn, P. (2012). Which experiences of health care delivery matter to service users and why? A critical interpretive synthesis and conceptual map. Journal of Health Services Research & Policy, 17(2), 70–78.
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review--a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10 Suppl 1, 21–34.

46. Questions?
Contact me for resources/information: