1. Systematic Literature Review: Step by step process
Adeel Tariq
Ph.D. Candidate
School of Management
Asian Institute of Technology

2. Outline: Outline Why literature review? How to get started
Systematic Literature Review Process

3. Question? What does a Literature Review mean to you ? Or
What do you understand about a systematic literature review?
Why do we need to do them?
Identify a gap
Inform your research
Help evaluate your research

4. In this study, 34% of examiners made critical comment on the literature review. The two ‘most common criticisms’ were the ‘failure to use recent literature and the inability to critically assess the existing literature’ (p. 179).

5. X has not engaged adequately with a body of material relevant to the core of the thesis. (500810150)
I didn’t find evidence of an advanced level of understanding of … theory as applied in … [X area] which is expected of a PhD candidate in this field. ( )

7. Why do them? Identify gaps Avoid re-inventing wheel Build on past work
Identify future colleagues
Increase your knowledge of the subject
Identify key works in your area
Provide intellectual context of your work
Identify opposing views
Demonstrate you can find and analyse research
Identify ideas & methods relevant to your work
Bourner, 1996

8. Literature Review: Why Literature Review
To Clarify Conceptual Issues
What is already Known about topic
What are origins and definitions of the topic?
What research question(s) are you asking? and Why?
Why Literature Review
What are Major issues and debates
How might you challenge existing beliefs or add to it
How knowledge on topic is structured?
What are the key theories, concepts & ideas

9. “Literature reviews …… introduce a topic, summarise the
The Literature Review ?
“Literature reviews …… introduce a topic, summarise the
main issues and provide some illustrative examples.”
This is an aspect of systematic reviews. What are your thoughts on this?
from
Agree? Disagree?

10. The Literature Review ?
Readers need to be able to judge whether all of the relevant literature is likely to have been found, and how the quality of studies was assessed.
1 Agree? 5 Disagree? 10

11. What is a systematic Literature review?
“A review that is conducted according to clearly stated, scientific research methods, and is designed to minimize biases and errors inherent to traditional, narrative reviews.”
Margaliot, Zvi, Kevin C. Chung. Systematic Reviews: A Primer for Plastic Surgery Research. PRS Journal. 120/7 (2007)
Ensures that all available evidence is taken into account and minimises ‘cherry-picking’
Not performing SRs can be dangerous
2nd point – minimise the chance of studies with +ve or –ve effects being cherry-picked to support a certain position.
3rd point – failure to conduct SRs in medicine can be very bad and I’ll give you an example why = corticosteroids in women in preterm delivery.

12. Why we need systematic reviews
Researchers and policy makers are inundated with unmanageable amounts of information
Over 20 million citations in PubMed
Approx. 75 to 100 RCTs published daily
Usually impossible to consider all relevant individual primary research studies in a decision making context

13. Systematic v.s. Narrative reviews
Agreed standards Context and differences
High degree of focus Bias of empirical studies
Minimize bias
Systematic reviews (SR) has disadvantages as well as advantages
Usually more than one reviewer so agreed standards and need to minimise bias. Therefore often quite narrow focus, e.g. find as much possible research relevant to question
However, not particularly useful if want more narrative approach that shows individuality or broader coverage.
Refer to ‘Balancing the strengths of systematic and narrative reviews’ (Collins & Fauser, 2004).
“For some review topics, however, the strengths of the SR may turn into weaknesses…do not allow for comprehensive coverage” but “narrative reviews do not reveal how the decisions were made about relevance of studies” p
So even if you’re using more typical narrative review, SR provides transparency and rigour that might help you with the process; help you to identify your sources and keep track for example 13 13

14. How to conduct a systematic literature review?
How will you answer this?
Conduct a trial?
Do a ceremonial dance to the great and mighty evidence gods?
Search and appraise a relevant SLR?

15. Steps
Formulating a problem/Question?
Locating and selecting studies
Critical appraisal of studies
Analyzing and presenting results
Synthesizing Results

16. What is your research question?
Start with the question
How you choose the study depends on your research question
How to get a question?
What is your research question?
1. Formulating a Problem

17. Does drinking coffee raise people’s blood pressure
(when compared to chocolate)?
SPQR
Focus Statement
Standard SR question contains
subject- intervention- outcome- comparator

18. 2. Finding Relevant Studies
How can you search until you know what you are looking for?

19. Literature Search
“A comprehensive and reproducible literature search is the foundation of a systematic review.”
Margaliot, Zvi, Kevin C. Chung. “Systematic Reviews: A Primer for Plastic Surgery Research.” PRS Journal. 120/7 (2007) p.1837

20. Searching

22. APHO staff conference 2009
Boolean operators 22

23. Publish or Perish:
Publish or Perish Demo

24. Phase 1- Identify the Research
a broad but defined, systematic sweep
Defined search terms – record recall and precision
Defined search arena - e.g. databases, citation indices, reference lists from primary and review articles, grey literature, conference proceedings, research registers, the internet, individual researchers/practitioners
Other broad search limits, e.g. language, date,
TIPS!
Document the
search protocol
and record what
research was
found
Systematically
manage the
search output,
e.g. using
endnote
Compile list of terms for each component, considering:
Synonyms (from different times and places)
Other words and phrases that are related to what you’re interested in
Words that are broader
Words that are more focused
Index terms or keywords

25. Phase 2- Selection
select from research using criteria related to your research question
Develop inclusion or exclusion statements, these might relate to study outcomes, research design, methods used, population worked with etc.
e.g. studies with a mixed population of men and women
e.g. random control trials only
e.g. maximum exposure time of 10mins
TIPS!
Document the
statements
and
their purpose
(might be
pragmatic or
research related)

26. Running the search How many titles and abstract can you check?
How easy will it be to decide to accept or reject a record?
Record the reason for rejection for “Excluded studies”

27. Don’t stop searching when you’ve stopped searching

28. Inclusion/Exclusion Criteria
“Once the study question is formalized, the authors must compose a comprehensive list of inclusion and exclusion criteria.”
“To avoid selection bias, inclusion and exclusion criteria should be agreed upon and formalized before data extraction and analysis.”
Margaliot, Zvi, Kevin C. Chung. “Systematic Reviews: A Primer for Plastic Surgery Research.” PRS Journal. 120/7 (2007) p.1836

29. Literature Search Challenges
Database Bias - “No single database is likely to contain all published studies on a given subject.”
Publication Bias - selective publication of articles that show positive treatment of effects and statistical significance.
Hence, it is important to search for unpublished studies through a manual search of conference proceedings, correspondence with experts, and a search of clinical trials registries.
Margaliot, Zvi, Kevin C. Chung. “Systematic Reviews: A Primer for Plastic Surgery Research.” PRS Journal. 120/7 (2007) p.1837

30. Literature Review Challenges (cont.)
English-language bias - occurs when reviewers exclude papers published in languages other than English
Citation bias - occurs when studies with significant or positive results are referenced in other publications, compared with studies with inconclusive or negative findings

31. 3. Critical appraisal of studies
“Assessing the quality of methodology is a critical part of the systematic review process”
No standard approach but there are hierarchies in fields of study

32. 3. Critical appraisal of studies
What would be appropriate
to consider when critically
appraising research in your area?
Different fields of study have different variables or ‘things’ that they critique and there are different priorities. For example, in health care random controlled trials very important. In history, source is key.
Activity – divide up into disciplinary areas. Brainstorm the criteria/variables/issues that you would evaluate, appraise, critique the literature on. Share with larger group to demonstrate hierarchies within fields.
Where do you look? 32 32

33. 3. Critical appraisal of studies
Authority
Validity
Accuracy
Objectivity
Currency - Recency
Coverage
Location
Different fields of study have different variables or ‘things’ that they critique and there are different priorities. For example, in health care random controlled trials very important. In history, source is key.
Activity – divide up into disciplinary areas. Brainstorm the criteria/variables/issues that you would evaluate, appraise, critique the literature on. Share with larger group to demonstrate hierarchies within fields.
Where do you look?
Student Learning Development, TCD 33 33

34. 3. Critical appraisal of studies
No standard approach but there are hierarchies in fields of study
disciplinary
perspective
connection
to findings
coverage
working
understanding
critical
appraisal
scholarliness
literature use

35. 4. Collect data & analyse Evaluate
Synthesize results of literature review
Tables to compare
Descriptive
Meta-analysis
There are several tools you could use to help you synthesise and evaluate the literature.
Use tables to compare e.g. across themes – show Hart example.
Use mapping to see patterns and relationships – show Cresswell example.
Use grids to compare across studies – see next slide. Idea is to have common template so being systematic.
This would be ‘Synthesise results of Literature review’ in Fink’s steps: “Report on current knowledge; justify the need for research; explain research findings; describe quality of research” (p. 105)
Descriptive review – qualitative synthesis of results
SR would most likely do meta-analysis of studies – statistical combination of results.
What to analyse/synthesise? Maybe:
Aims and objectives of studies
Central thesis/argument
Outcomes
Theoretical framework
Context and background
Research design and method
Findings
Contribution to field
As you read the literature take notes from each source based on these (or any other relevant factors) and get into the writing habit
Student Learning Development, TCD 35 35

36. Example of grid or checklist to compare studies
Author
Year
Aim
Methods
Conclusions
Critique
Example of grid or checklist to compare studies
Student Learning Development, TCD 36 36

37. Write up literature review - Structure
Background
Purpose/Research question
Method
Findings
Discussion
Implications/Recommendations
This is one example of outline of a literature review (which might be your lit review chapter)
Here, method refers to how you conducted your literature review, not how you did your research.
Findings/Discussion refers to analysis of retrieved literature
Implications/Recommendations might be what this means for your thesis. For example, a review of literature on learning skills interventions revealed: 1) two different theoretical approaches that underpin interventions and 2) very little qualitative research on these interventions; therefore, thesis research will focus on these aspects.
Read other theses 37

38. Remember Not a description of what you found
Not a list of relevant research
Your review must be directed by your research question

39. Most important thing: Be organised!!!

40. Gantt charts

41. Voice Writer as tour guide - voice
Tell the reader what knowledge and ideas there have been on a topic,
Assess them – what are their strengths and weaknesses
Writer as tour guide - voice

42. What makes a review “Systematic”?
1. PICO – which you will be familiar with now
2. Using reproducible methods including predefined search strategies
3. A-priori defined inclusion/exclusion criteria and selection, extraction etc. described
4. Often using a mix of objective and subjective assessments
5. This can include meta-analysis and assessment of heterogeneity
6. Not extrapolating beyond the data or populations that were studied

43. Figure 1: Yearly and cumulative publications on green product and process innovation

44. Conceptual Framework: