EBC course 10 April 2003 Critical Appraisal of the Clinical Literature: The Big Picture Cynthia R. Long, PhD Associate Professor Palmer Center for Chiropractic Research

Outline Overall purpose of critical appraisal Consider “levels of evidence” Some tips on interpreting “significant” findings Potentially useful references

Overall Purpose To critically appraise a research article in order to assess the validity of the authors’ conclusions

Sections of an Article Introduction: Background and explanation of rationale for study. Methods: How was study done? Should allow study to be replicated. Results: Report results (data). Discussion: Interpret results. Draw conclusions from results. Abstract: Article summary.

Recommendation Read the abstract last when familiarizing yourself with critically appraising the literature. Assess evidence from reading the article, not the abstract. Read the abstract in deciding whether or not you are interested in the topic of the article.

Critical Appraisal similar across types of studies for Introduction and Discussion sections information in Methods sections may differ information in Results sections may differ

Overall Questions to Ask Is the study design appropriate to address the research question? In the Discussion Section: Are the findings... –...consistent with the research question of the study?

Terminology: Types of Clinical Studies Risk Diagnostic Prognostic Intervention

Terminology: Clinical Intervention Studies Evaluates which treatment interventions are most useful and effective for a given clinical condition

Clinical Intervention Studies One-group Pre/post Study Pilot RCT Randomized Clinical Trial (RCT) Systematic Review of RCTs

Levels of Evidence

Is the study design appropriate to address the research question? In the Discussion Section: Are the findings... –...consistent with the research question of the study?

One-group pre/post design Addresses: do patients improve after treatment? Can’t address: is treatment effective for patients? –No comparison group –Patients may improve: natural course of condition, in study, change in lifestyle, treatment

One-group pre/post design : Example Hawk C, Long CR, Azad A. Chiropractic Care for Women with Chronic Pelvic Pain: A Prospective Single-Group Intervention Study. JMPT 1997;20: –Results: women improved! –Conclusion: needs further study

Pilot RCT Addresses: is the RCT feasible? Can’t address: is treatment effective for patients? –Not powered (i.e. sample size not large enough) Note: often only published if RCT is determined to be unfeasible

Pilot RCT : Example 1 Hawk C, Long CR, et al. Issues in planning a placebo-controlled trial of manual methods: Results of a pilot study. J of Alt Comp Med 2002;8: –Results: recruitment not feasible; standardization of treatment protocol difficult among multiple sites –Conclusion: put on hold

Pilot RCT : Example 2 Bronfort G, et al. Nonoperative treatments for sciatica: A pilot study for a randomized clinical trial. JMPT 2000;23: –Results: recruitment not feasible! –Conclusion: put on hold

RCT Addresses: is the treatment effective for patients? Caveats: –Must be a powered study sample size must be formally justified in the Methods section (based on: effect size— minimally important clinical difference; variability of outcome measure; statistical test)

RCT : Example Hurwitz E, et al. A randomized trial of chiropractic manipulation and mobilization for patients with neck pain: Clinical outcomes from the UCLA neck-pain study. Am J Public Health 2002;92: –Results: no statistically or clinically significant differences among groups –Conclusion: cervical spine mobilization is as effective as manipulation in reducing neck pain and related disability among chiropractic patients

Overall Questions to Ask In the Discussion Section: Are the findings... –...consistent with the research question of the study? –…consistent with the results presented? –…given in the context of current evidence?

Systematic Review of RCTs Addresses: is the treatment effective for patients? Looks at all RCTs of the treatment for patients and combines based on quality of original RCTs

RCT : Example Bronfort G, et al. Efficacy of Spinal Manipulation for Chronic Headache: A Systematic Review. JMPT 2001;24: –Conclusions: can’t make firm conclusions (few trials of adequate methodological quality)

RCT : Example –Conclusions: Cervicogenic: SMT better effect than massage Tension-type and migraine: SMT effect comparable with commonly used first-line prophylactic prescription meds

Terminology: Descriptive Statistics (Results) Patient characteristics at baseline –Examples: mean, standard deviation, median, range, percentage –Assess group comparability on baseline characteristics Assess generalizability of results to target population

Terminology: Analytical Statistics (Results) Assess statistical significance with confidence intervals and p-values –Within and between group differences –Make inference about target population Must be appropriately interpreted in the context of the research question and the study design

Terminology P-values and confidence intervals: –Reflects measure of effect relative to variation and sample size

Statistical Significance  Interpreting p-values:  p<0.01  statistically significant difference!  0.01  p  0.05  statistically significant difference  0.05<p  0.10  borderline statistically significant difference  p>0.10  no statistically significant difference

“Significant” Findings jargon term… Need to consider BOTH statistical and clinical significance

Clinical Significance i.e. clinical importance is defined before study is conducted assessed with descriptive statistics (e.g. mean improvement in outcome measure)

Possible Scenarios statistically and clinically significant findings clinically significant, but not statistically significant statistically significant, but not clinically significant

Critical Appraisal References Useful? It depends… BMJ series available for free on bmj.com JAMA articles? Chiropractic Research Review