Gerald Dyer, Jr., MPH October 20, 2016 HETEROGENEITY Gerald Dyer, Jr., MPH October 20, 2016

OVERVIEW P&R CHAPTER 7 (Up to Page 217) Chapter 15 Chapter 16 Chapter 18 (Brief Quiz Game)

HETEROGENEITY??

YOUR FIRST THOUGHTS?

DEFINITION Heterogeneity is defined in the text as “we use heterogeneity to mean heterogeneity in true effects only.” pg. 106 Simply Complex Includes Several Measures Maybe more than just vertical and horizontal lines…

WHAT IS HETEROGENIETY?? -_-

2 Types of Heterogeneity P&R Chapter 7 2 Types of Heterogeneity Differences in studies by methods, participants, and other unknown sources. Differences in studies due to their quantitative findings, known as statistical heterogeneity. Systematic reviews make efforts to limit heterogeneity by methods and participants by utilizing very precise inclusion criteria. However, some use broad inclusion criteria; therefore, heterogeneity is expected. Stat heterogeneity can be due to differences in baseline characteristics of the population and methodological differences.

EXPECT CONSIDERABLE HETEROGENEITY P&R Chapter 7 Primary Study Complexities Social interventions are complex from the content of the intervention, study population, outcomes, effectiveness of intervention, and variation of the intervention while being implemented. EXPECT CONSIDERABLE HETEROGENEITY

SOCIAL HETEROGENEITY VARIABILITY Study populations, interventions, and settings Outcomes Study Designs “Social heterogeneity may incorporate not only socio-demographic and individual differences, but also, historical, cultural, spatial, and other differences, which may affect both the delivery and impact of interventions being reviewed.” (See Page 216)

A META-ANALYST’S INTEREST If it is greater than chance, then the results suggest that the studies may not be similar enough to permit comparisons. P&R describe the Q statistic and the I^2 statistic. The Q stat can test for heterogeneity, but it is considered low power. Even when there is no evidence of stat heterogeneity, a meta-analysis may not be appropriate because similar effect sizes may be obtained from studies that are conceptually different. The I^2 stat was developed to detect the degree of inconsistencies between studies. It should be noted that the quantification of heterogeneity is only one component of a wider variability across studies. INTERESTED IN WHETHER STATISTICAL HETEROGENEITY IS GREATER THAN CHANCE

CHAPTER 15 Central theme: “The goal of a synthesis is not simply to compute a summary effect, but rather to make sense of a pattern of effects.” Problem addressed: “The observed variation in the estimated effect sizes is partly spurious.” Array of dispersion measures: Q Statistic, p-value, T2, T, and I2

CHAPTER 16

ISOLATING THE VARIATION IN TRUE EFFECTS “When we speak about the heterogeneity in effect sizes, we mean the variation in the true effect sizes.” (TEXT page 108) Extraction Mechanism of true between-studies variation from the observed variation Compute the total amount of study-to-study variation observed. Estimate how much the ovserved effects would be expected to vary from each other if the true effect was actually the same in all studies. The excess variation (if any) is assumed to reflect real differences in effect size (heterogeneity).

Q (Part 1) “Statistic that is sensitive to the ratio of the observed variation to the within-study error.” TEXT page 109 Q is a standardized measure, and it is determined on the assumption that all studies share a common effect size. Q depends on the degrees of freedom: df = k - 1

Q (Part 2) Q is not an intuitive measure. It is a sum and depends strongly on the number of studies. Q is on a standardized scale, but it is used to test the assumption of homogeneity. Q can be used to calculate T, T2, and I2 SEE FIGURE 16.3 on TEXT PAGE 111

Q - Conclusion Concluding Remarks: A significant p-value provides evidence that the true effects vary. A non-significant p-value should not be taken as evidence that the effect sizes are consistent. The test assesses the viability of the null hypothesis (all studies share a common effect size) and not to estimate the magnitude of true dispersion.

TAU-SQUARED (T2) Defined as the estimate of the variance of the true effect sizes T2 reflects the absolute variation and is in the same metric (squared) as the effect size itself. T2 is used to assign weights under the random-effects.

T (Tau) Estimate for the standard deviation. T can be used to describe the distribution of effect sizes about the mean effect. Note that if we wanted to make predictions about the distribution of true effects we would need to take account of the error in estimating both the effect size and T. See Page 117

I2 I2 allows us to discuss the amount of variance on a relative scale. Example: After determining what proportion of the observed variance is real from I2, we can know is there is any explanation for the variance. I2 ~ 0 -> There is nothing to explain. Values of 25%, 50%, and 75% might be benchmarks for low, moderate, and high, respectively.

Let’s Play a Brief, Little Game! =) CHAPTER 18 Let’s Play a Brief, Little Game! =)

NAME!!!

THAT!!!

HETEROGENEITY!!! HETEROGENEITY!!!

NAME THAT HETEROGENEITY? See Figure 16.4 on Page 113 Focus on Plots A and C Impact of Q and study number. What are the trends in the impact of Q? Answer: Additional precision found in plot C makes the p-value move away from zero, compared to A. The p-value for the columns moved toward 1.0 as we added studies. What do the degrees of freedom mean in this example? Since Q is less than the degrees of freedom, the additional evidence strengthens the case that the excess dispersion is zero. What assumptions should we be cautious of when interpreting the Q and p-values? -A significant p-value provides evidence that the true effects vary. -A non-significant p-value should not be taken as evidence that the effect sizes are consistent. -The test assesses the viability of the null hypothesis (all studies share a common effect size) and not to estimate the magnitude of true dispersion.

THANK YOU!!