Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2006 1 Literature review designs Narrative Review Systematic Review Meta-analysis.

Similar presentations

Presentation on theme: "© 2006 1 Literature review designs Narrative Review Systematic Review Meta-analysis."— Presentation transcript:

1 © Literature review designs Narrative Review Systematic Review Meta-analysis

2 © Evidence-based Chiropractic 2 Literature review Defined as a systematic, explicit, and reproducible way of identifying, evaluating, and interpreting all of the research findings and scholarly work available on a topic –A high-quality review is not haphazard –Ideally, all of the existing work should be included Considered descriptive or observational

3 © Evidence-based Chiropractic 3 Meta-analysis  Quantitatively combines the results of studies that are the result of a systematic literature review.  Capable of performing a statistical analysis of the pooled results of relevant studies. The three types of literature review designs Narrative Review  Selective review of the literature that broadly covers a specific topic.  Does not follow strict systematic methods to locate and synthesize articles. Systematic Review  Utilizes exacting search strategies to make certain that the maximum extent of relevant research has been considered.  Original articles are methodologically appraised and synthesized.

4 © Evidence-based Chiropractic 4 Narrative reviews Summarize in general what is in the literature on a given topic –Often written by experts in a given field –A good source for background information Do not follow strict systematic methods like the other literature review designs –Therefore, they are prone to bias –Lower in the hierarchy of evidence

5 © Evidence-based Chiropractic 5 Narrative reviews (cont.) Authors like to write them because they are relatively easy to carry out and compose Practitioners like to read them because they are easier to comprehend than more complex designs –Do not require a lot of background knowledge to understand the message

6 © Evidence-based Chiropractic 6 Narrative reviews are prone to bias They do not employ many of the safeguards needed to control against bias –Authors may be selective as to which articles are included –They may include articles that support their hypothesis and exclude those that do not –Rigorous appraisal methods are not used to evaluate included articles

7 © Evidence-based Chiropractic 7 Prone to bias (cont.) During the literature search –Authors have their own opinions on the topic and may try to find studies that support their viewpoint and overlook conflicting studies During the synthesis of the literature –The approach to analyzing the collected information is often subjective and disorganized

8 © Evidence-based Chiropractic 8 Prone to bias (cont.) In the discussion and conclusion –The authors’ opinions may be mixed together with evidence –Authors may simply count the number of studies on each side of an issue and then espouse the view presented by the majority without considering the strength of each study e.g., quality, research design used, the effect size, and sample size

9 © Evidence-based Chiropractic 9 Selection bias in narrative reviews a.k.a., reference bias Occurs when authors choose articles that support their own conclusions and exclude articles with conflicting views –Results in an erroneous representation of the literature –The review may lead undiscerning readers astray

10 © Evidence-based Chiropractic 10 Systematic reviews Use strict methods to locate, appraise and synthesize all research on a topic –Similar to narrative reviews, but with improved procedural quality Designed to answer specific clinical questions where several primary studies exist –Thus, a good source of clinical evidence

11 © Evidence-based Chiropractic 11 Systematic reviews (cont.) Articles are evaluated using appraisal instruments –In an attempt to achieve impartiality –More reproducible Dissimilarities between the findings of studies are investigated Multiple reviewers are usually involved –Any disagreements are resolved

12 © Evidence-based Chiropractic 12 Systematic reviews (cont.) The search and selection criteria for articles are well defined –Multiple databases should be searched –Explicit inclusion criteria The results of the included studies are qualitatively or quantitatively synthesized –Qualitative – written information is merged –Quantitative – data are merged

13 © Evidence-based Chiropractic 13 Systematic reviews (cont.) Because of strict methodology and thoroughness, conclusions are typically less biased than narrative reviews Nevertheless, they may still be influenced by the authors’ opinions –Still a potential for selection bias –Criteria may be applied differently when appraising included studies –Therefore, must be critically appraised

14 © Evidence-based Chiropractic 14 Systematic reviews (cont.) No widely accepted method exists for assessing the validity of studies –i.e., the process is a judgment call –Thus, reviewers sometimes disagree Information derived from a systematic review may be too narrow to completely answer a specific clinical question –Given that they are designed to answer focused questions about patient care

15 © Evidence-based Chiropractic 15 Differences between narrative and systematic reviews FeatureNarrative ReviewSystematic Review TopicTypically broad-scopedFocused research question Data sources and search strategy The search strategy and databases that were used may not be provided The search strategy is explicit and comprehensive with a list of all databases that were utilized AuthorshipA recognized expert(s) on the topic A team of experts having methodologic and clinical expertise Article selection criteria Typically not specifiedConsistently applied inclusion and exclusion criteria SearchingMay be extensive, intended to locate literature on the topic area in question Extensive, intended to locate all primary studies on a particular research question Appraisal of included articles Indefinite, may be variableCritical appraisal is meticulous, typically involving the use of data extraction forms SynthesisA qualitative summary is usually provided A qualitative summary is provided, quantitative when the data can be pooled InferencesSometimes evidence-basedUsually evidence-based

16 © Evidence-based Chiropractic 16 Systematic reviews typically only include RCTs Studies are included primarily based on their quality –Most systematic reviews only include RCTs because it is the only design that adequately controls for confounding variables and biases –The potential for studies to overrate the treatment effect is higher when bias is present Studies using other less rigorous designs are usually eliminated

17 © Evidence-based Chiropractic 17 Conducting systematic reviews

18 © Evidence-based Chiropractic 18 Search strategy Should be described in enough detail so that another researcher could replicate the results, including: –Database(s) searched –Date the search was performed –Time-frame encompassed by the search –A list of search terms used –Languages

19 © Evidence-based Chiropractic 19 Search strategy (cont.) Conference proceedings, unpublished studies, and hand-searching of journals are sometimes included in the search –If so, the procedures involved should be described –Authors must justify using unpublished work

20 © Evidence-based Chiropractic 20 Weighting of studies Articles may be rejected in a systematic review due to their poor quality Alternatively, studies are assigned weights in relation to their assessed validity –Studies that are more valid will have more influence on the review’s final results –Based on methodological quality, width of the confidence intervals, and external validity

21 © Evidence-based Chiropractic 21 Publication bias Studies with statistically significant results are more likely to get published than those with non-significant results Causes of publication bias: –The author or funding source does not consider a “failed study” worthy of submission –Journals are less likely to publish studies that fail to show positive results

22 © Evidence-based Chiropractic 22 Publication bias (cont.) Reviews affected by this bias tend to give an overoptimistic view of the effectiveness of the therapy –The chance of this bias occurring is reduced when authors of systematic reviews search sources other than journals Publication bias in situ –A type of bias where a portion of a study’s results are suppressed

23 © Evidence-based Chiropractic 23 Meta-analysis a.k.a., quantitative systematic review A type of systematic review that statistically combines the results from a number of studies Capable of producing a single estimate of the effect of a treatment –Represents the “average” treatment effect –An estimate of the true treatment effect size

24 © Evidence-based Chiropractic 24 Meta-analysis (cont.) The same explicit methods as systematic reviews are utilized Systematic reviews and meta-analyses are are at the top of the hierarchy of evidence because of their strict methodology

25 © Evidence-based Chiropractic 25 Weighted average Meta-analyses typically produce a weighted average for the treatment effect estimate –Small samples are more susceptible to chance variations than larger studies –Thus, they are given less weight than larger studies so they will have less influence on the final estimate

26 © Evidence-based Chiropractic 26 Weighting (cont.) Weighting is also based on study quality –The quality of the individual studies is rated and resulting numeric scores are calculated –A corresponding weight is assigned for each study prior to analysis

27 © Evidence-based Chiropractic 27 Meta-analyses can increase power Data from individual studies are combined, which in effect increases sample size –Chiropractic studies commonly involve too few subjects to detect true differences between the groups –Pooling data reduces the potential for type II error –More likely to detect a treatment effect, if there actually is one

28 © Evidence-based Chiropractic 28 Homogeneity and heterogeneity Homogeneity –Similarities of included studies that allow them to be compared –Homogeneity is preferred in meta-analyses –Achieved by using suitable inclusion criteria Heterogeneity –Dissimilarities of studies that hamper or even prevent a realistic comparison of studies

29 © Evidence-based Chiropractic 29 Factors that contribute to heterogeneity Heterogeneity in the study samples –Caused by conflicting inclusion and exclusion criteria, differences in patients’ baseline health status, dissimilar geographical locations of groups, etc. Heterogeneity in the study design –E.g., the way dropouts were managed in the statistical analysis or the length of time allowed for patient follow-up

30 © Evidence-based Chiropractic 30 Factors that contribute to heterogeneity (cont.) The way patients were handled –Regarding comorbid conditions, handling of complications, the control practitioners had in patient care, or the outcome measures used Statistical heterogeneity –When the observed treatment effects of studies are more dissimilar than what would be expected by chance

31 © Evidence-based Chiropractic 31 Consequences of heterogeneity When the results of studies in a meta- analysis are inconsistent, it reduces confidence in its conclusions The meta-analysis may actually be worthless if too dissimilar –For instance, combining studies that used different types of comparison groups –Or outcomes that were dissimilar

32 © Evidence-based Chiropractic 32 Forest plot A type of graph often used in meta- analyses to illustrate the treatment effect sizes of the studies Each study is represented by a black square that is an estimate of their effect sizes A horizontal line extends to either side of the squares, the 95% confidence interval

33 © Evidence-based Chiropractic 33 Forest plot

34 © Evidence-based Chiropractic 34 Interpreting a forest plot If a study’s 95% CI crosses over the vertical line, it is not statistically significant A diamond with a CI line is sometimes presented at the bottom of the forest plot to represent an overall estimate The black squares may vary in size representing the weights of the studies

35 © Evidence-based Chiropractic 35 Weighting and overall effect

36 © Evidence-based Chiropractic 36 Effect size The difference between the means of the treatment and control groups When studies are combined in a meta- analysis, the units of measurement are not always comparable Effect sizes are standardized to resolve this problem producing the standardized mean difference

37 © Evidence-based Chiropractic 37 Standardized mean difference The effect size divided by the pooled standard deviation –Pooled standard deviation has been adjusted for the differences in the sizes of the groups Represents the standardized difference between group means –i.e., the relative magnitude of the experimental treatment

38 © Evidence-based Chiropractic 38 Cohen’s d

39 © Evidence-based Chiropractic 39 Odds ratio (OR) Cohen’s d is appropriate with continuous data An OR is appropriate when the study’s outcome measure is dichotomous –e.g., pain versus no pain OR is a comparison of the odds of the outcome being present in the treatment group against the control group

40 © Evidence-based Chiropractic 40 Relative risk (RR) a.k.a., risk ratio A comparison of the risk of having the outcome in the treatment group with that of the control group Sometimes OR is reported and sometimes RR –Experts do not agree on which is most appropriate

41 © Evidence-based Chiropractic 41 OR vs. RR Consider a hypothetical lower back pain study with 25 patients in each group 5 in treatment group and 10 in control group are still in pain at the study’s end OR = = =.38 RR = = =.5 5/20 10/ /25 10/25.2.4

42 © Evidence-based Chiropractic 42 Meta-analyses are most valid with RCTs However, about half of meta-analyses include observational studies –Primarily cohort and case-control Observational studies are much more susceptible to biases and confounding than RCTs –Therefore, it is usually inappropriate to statistically combine the results of such studies

43 © Evidence-based Chiropractic 43 Subgroup analysis Meta-analyses typically include patients with a variety of characteristics –e.g., age, gender, condition severity, patient history, etc. Patients in these subgroups may respond to treatment differently –e.g., low back pain patients with leg pain may respond to treatment differently than low back pain only patients

44 © Evidence-based Chiropractic 44 Subgroup analysis (cont.) Carried out to identify variation between patient groups regarding certain outcomes or findings The process helps readers to distinguish the effects of a treatment between subgroups The statistical power of the subgroups’ will decline as a result

45 © Evidence-based Chiropractic 45 Meta-regression A statistical procedure that adjusts for differences between studies in meta- analyses –May be used in subgroup analyses Similar to simple regression –Predictor variables: characteristics of the studies –Outcome variable: treatment effect estimate

46 © Evidence-based Chiropractic 46 Sensitivity analysis A type of subgroup analysis that considers non-patient characteristics, e.g., treatment variations or study methodology Determines the extent heterogeneity affected the results of a meta-analysis –If the results are weak, sensitivity analysis may reveal significant treatment effects when different methods are used

47 © Evidence-based Chiropractic 47 Narrative versus systematic reviews and meta-analyses There are no strict rules regarding the creation of either type –Therefore, it may be difficult to decide if a given review is systematic or narrative Narrative reviews do not typically use systematic methods –They tend to be subjective and prone to bias –Cover broader topics than systematic reviews

48 © Evidence-based Chiropractic 48 Narrative vs. systematic reviews (cont.) May be unclear how conclusions were drawn from the data in narrative reviews –Often the number of studies supporting one side of a topic is counted and then compared with the number supporting the opposite side –The side with the highest number of supporting articles wins –This process does not consider the weight of studies as in systematic reviews

49 © Evidence-based Chiropractic 49 Advantages and disadvantages of narrative versus systematic reviews AdvantagesDisadvantages Narrative reviews Present a general overview covering a specific topic which provides primary information and/or an update Fairly easy for novice authors to prepare May not provide the best available answers to focused questions Findings are less reliable Systematic reviews Present a comprehensive review of the literature based on all available research with regard to a focused research question Provide an estimate of the “true” answer to the research question Specialized expertise of reviewers is required Involve a formal research protocol Findings are only relevant to a single question

50 © Evidence-based Chiropractic 50 QUOROM Statement QUOROM (Quality of Reporting of Meta- analyses) The QUOROM Statement w as developed to reduce the potential for reviews to reach contradictory conclusions An attempt to ensure uniformity and accurate reporting Has been adopted by many journals

51 © Evidence-based Chiropractic 51 Structure of review articles Should conform to the anatomy of a typical scholarly article –i.e., Abstract, Introduction, Methods, Results, Discussion, and References Literature reviews are in reality a type of research –However, conclusions are derived from original sources of information

52 © Evidence-based Chiropractic 52 Structured Abstract Objective –The author should clearly state the purpose of the article Background –A description of what prompted the review –Presentation of a context for the review Methods –A description of the methods used

53 © Evidence-based Chiropractic 53 Structured Abstract (cont.) Discussion –The implication and relevance of the information the review presents Conclusion –Summary of what the review contributes to the literature –What new conclusion can be drawn as a result of the synthesis of the literature

54 © Evidence-based Chiropractic 54 Introduction Presents the background and context of the problem that inspired review The topic should be clearly defined, and tied in with clinical practice Novel terms should be defined A description of the course of the disease, common outcomes and treatment options A synopsis of existing research

55 © Evidence-based Chiropractic 55 Introduction (cont.) The importance and need for the review should be established by showing –That there are gaps in the literature on the topic –The extent of the condition’s negative impact on society in terms of human suffering and monetary costs –Explained in enough detail to substantiate the need for the review

56 © Evidence-based Chiropractic 56 Introduction (cont.) A focused and well-constructed question should be present –Provides direction for the review –Assists readers in determining if the review is applicable to their individual clinical circumstances –Should help establish the review’s inclusion criteria

57 © Evidence-based Chiropractic 57 Methods Describes the search process and strategies involved, including: –Databases searched –Search terms –Search limits e.g., publication years, languages, ages, etc. Should include enough detail to enable others to replicate the search

58 © Evidence-based Chiropractic 58 Methods should include The criteria that were used to include or exclude studies –For example, exclude surgery related studies or drug trials A description of how studies were appraised –Rating instruments are typically used –However, the reliability of these instruments varies considerably

59 © Evidence-based Chiropractic 59 Methods should include (cont.) Information about –How the relevance of primary studies was ascertained –How the data were extracted and synthesized –Sources of heterogeneity How much How it was handled

60 © Evidence-based Chiropractic 60 Results The outcome of the search process is presented Including information on –The number of articles retrieved –How many articles were excluded from the review and which of the inclusion criteria they failed to meet Look for evidence of selective referencing

61 © Evidence-based Chiropractic 61 Results (cont.) The Results section is sometimes very short –When short, details about the retrieved articles are provided in the Discussion section However, it may be longer –The characteristics of the included studies may be described and contrasted in this section –Often presented in tables

62 © Evidence-based Chiropractic 62 Discussion The findings of all of the articles in the review are synthesized to generate a conclusion –There may or may not be a separate Conclusions section Information about the etiology, pathophysiology, diagnosis, treatment, and prognosis of the condition at issue is often provided

63 © Evidence-based Chiropractic 63 Discussion (cont.) Presents a new perspective on the topic that is usually more reliable than any of the individual articles in the review Caution – authors have the leeway to defend articles that support their viewpoint and challenge those that do not –Systematic methods control for much of this subjectivity, but it is still possible in the best types of reviews

64 © Evidence-based Chiropractic 64 Discussion (cont.) The synthesis is the heart of the literature review design –Consequently it is important to ensure that a meaningful integration is accomplished The author should offer an interpretation of the literature reviewed –A critical appraisal of the articles reviewed may be in order

65 © Evidence-based Chiropractic 65 Discussion (cont.) The conclusion should be in agreement with the evidence presented in the review Authors should emphasize what new information can be gained The conclusion should not merely repeat what was previously written

66 © Evidence-based Chiropractic 66 References Should be comprehensive and cite all articles included in the review Derived almost entirely from peer- reviewed journals –But may include conference proceedings, textbooks, and government documents –Unpublished works too; but keep in mind, they have not been peer-reviewed

67 © Evidence-based Chiropractic 67 References (cont.) No padding of references –Only enough articles should be cited to make a point Be alert for authors who misrepresent sources, which is a fairly common occurrence –e.g., misquote, selectively quote, omit relevant information, etc. –May have to read source material to detect

68 © Evidence-based Chiropractic 68 Appraisal of review articles Recent reviews are typically more useful because they contain the latest research –However, even the latest review may be behind the most current research The literature search should involve multiple databases The search strategy should be clearly described, along with the criteria used for study inclusion

69 © Evidence-based Chiropractic 69 Appraisal of reviews (cont.) Was a clear study question asked? –Reviews should have a clear purpose in the form of a study question or statement of purpose Especially with systematic reviews, but also applies to narrative reviews –Otherwise, the review will lack direction and may generate faulty conclusions

70 © Evidence-based Chiropractic 70 Appraisal of reviews (cont.) Was the study question focused on a specific clinical issue? –The study question must be focused on a specific clinical issue that is relevant to patient care to help practitioners make clinical decisions –Broad topic reviews provide background information, but are not very helpful in answering clinical questions

71 © Evidence-based Chiropractic 71 Appraisal of reviews (cont.) Were the inclusion/exclusion criteria that were used to select articles provided and were they appropriate? –Selection criteria should be appropriate for the clinical question that prompted the search –Difficult to assess the breadth of the review or judge the extent of the author’s partiality without this information

72 © Evidence-based Chiropractic 72 Appraisal of reviews (cont.) Was an adequate literature search conducted? –The search should be capable of finding all relevant studies –Appropriate databases and search terms –Citations should be collected from the reference sections of the included articles –How likely is it that relevant studies were missed during the search?

73 © Evidence-based Chiropractic 73 Appraisal of reviews (cont.) Was the validity of the studies included in the review assessed? –Reviews that include articles without appraising them are poorer quality and less reliable –Authors may examine issues such as the methods of randomization and blinding, whether concealment of the allocation to groups was used, etc.

74 © Evidence-based Chiropractic 74 Appraisal of reviews (cont.) Were the studies appraised in a dependable manner? –Should be reviewed by more than one reviewer to strengthen confidence in their findings –Reliable appraisal instruments should be used because they yield more consistent results Often included in an appendix

75 © Evidence-based Chiropractic 75 Appraisal of reviews (cont.) Were the included studies randomized trials? –Strengthens the usefulness of a literature review considerably when RCTS are involved –Reviews that include observational studies provide little evidence to support treatment –However, such reviews are helpful with patient care and qualify as the best evidence when observational research is all that exists

76 © Evidence-based Chiropractic 76 Appraisal of reviews (cont.) Were the results of the included studies similar? –Heterogeneity of studies may occur due to: Clinical factors (e.g., dissimilar participants, interventions or outcomes) Methodological factors (e.g., sample size or method of randomization) Contradictory results just by chance –Be concerned when discrepancies are large

77 © Evidence-based Chiropractic 77 Appraisal of reviews (cont.) Do the results of the synthesis logically flow from the studies that were included? –Authors have a great deal of latitude in the way they interpret and judge articles –Studies should be weighted based on study size and validity Reviews that simply compare the number of positive findings to the number with negative findings may be misleading

78 © Evidence-based Chiropractic 78 Appraisal of reviews (cont.) If the review dealt with therapy, was an estimate of treatment effect provided? –This is the overall estimate of treatment effect size in a meta-analysis –The width of the associated confidence interval reveals the precision of the estimate –A wide confidence interval degrades precision and lowers the degree of certainty regarding the review’s findings

79 © Evidence-based Chiropractic 79 Appraisal of reviews (cont.) Do the results of the review help with the care of patients? –In order to effectively use the results of a review in the care of a specific patient, the populations should be similar –The outcomes used in the review should be clinically important –Comparable patient and review interventions e.g., are manipulation and mobilization equal

80 © Evidence-based Chiropractic 80 Appraisal of reviews (cont.) Were directions for future research offered? –Especially helpful in reviews because authors assess the sum of all research on a topic and are in a very good position to determine what types of studies should be carried out in the future

Download ppt "© 2006 1 Literature review designs Narrative Review Systematic Review Meta-analysis."

Similar presentations

Ads by Google