Meta-Analysis: Synthesizing the evidence

Reminders Exam #3 (next class): Not cumulative, standard 40- question multiple choice Make-up LA and Paper rewrites (optional) Make-up Exam (optional): Short essays (60pts) Pick 3 of 4: Broad topics relevant to an entire unit (e.g., clinical research) or across units (e.g., validity) Short answer (100pts) Pick 8 of 12: Representative narrower topics (e.g., key considerations in designing RCTs)

Take Home Messages of the Course Understand the value of research and experience a greater desire to engage in research in some capacity Understand p-values Understand the difference between a true “experiment” and a non-experiment Gain a greater tolerance for ambiguity Improve scientific writing and statistical skills Understand different methodologies as tools that can improve the quality of research Gain practical skills/accomplishments to foster academic and career goals Live a more science-driven life

Today’s Overview What does it mean to synthesize evidence? Why would we want to synthesize evidence? What is a meta-analysis?

Weighing and Synthesizing Evidence Any problem in life can be informed by scientific evidence Interpersonal dynamics, parenting, studying, worker satisfaction, psychotherapy, medical treatments, educational interventions, traffic safety, health A so-called science-driven life would use scientific evidence to make key decisions Problem: Studies conflict (importance/size of effect, direction of effect, confidence/p-value), and any particular study has strengths and weaknesses (the many reliabilities and validities)

Examples

Weighing and Synthesizing Evidence Literature review Systematic literature review Meta-analysis Statistical approach for combining results from multiple studies Typically requires a systematic literature review to avoid bias Produces an average effect size (e.g., r, d, %) Bigger N  More power  More precision (e.g., better p-value, narrower confidence intervals)

Examples Mike’s current meta-analysis (separate file) Smith and Glass (1977). Meta-analysis of Psycho- therapy Outcomes.

Examples Hauch et al. (2016). Does training improve the detection of deception?

Examples Roberts et al. (2006). Patterns of mean-level change in personality traits across the life course.

Strengths and Limitations Excellent for synthesizing evidence, providing a starting point in a literature review, guiding policy Limitations Concern about publication bias and other biases in the systematic review process Concern about combining studies, samples, and interventions that are too heterogeneous