Statistical Analysis of the Two Group Post-Only Randomized Experiment

Analysis Requirements l Two groups l A post-only measure l Two distributions, each with an average and variation l Want to assess treatment effect l Treatment effect = statistical (i.e., nonchance) difference between the groups RXORORXORO

Statistical Analysis

Control group mean

Statistical Analysis Control group mean Treatment group mean

Statistical Analysis Control group mean Treatment group mean Is there a difference?

What Does Difference Mean?

Medium variability

What Does Difference Mean? Medium variability High variability

What Does Difference Mean? Medium variability High variability Low variability

What Does Difference Mean? Medium variability High variability Low variability The mean difference is the same for all three cases.

What Does Difference Mean? Medium variability High variability Low variability Which one shows the greatest difference?

What Does Difference Mean? l A statistical difference is a function of the difference between means relative to the variability. l A small difference between means with large variability could be due to chance. l Like a signal-to-noise ratio. Low variability Which one shows the greatest difference?

What Do We Estimate? Low variability

What Do We Estimate? Low variability Signal Noise

What Do We Estimate? Low variability Signal Noise Difference between group means =

What Do We Estimate? Low variability Signal Noise Difference between group means Variability of groups =

What Do We Estimate? Low variability Signal Noise Difference between group means Variability of groups = = X T - X C SE(X T - X C ) __ __

What Do We Estimate? Low variability Signal Noise Difference between group means Variability of groups = X T - X C SE(X T - X C ) = = t-value __ __

What Do We Estimate? l The t-test, one-way analysis of variance (ANOVA) and a form of regression all test the same thing and can be considered equivalent alternative analyses. l The regression model is emphasized here because it is the most general. Low variability

Regression Model for t-Test or One-Way ANOVA y i =  0 +  1 Z i + e i

Regression Model for t-Test or One-Way ANOVA y i = outcome score for the i th unit  0 =coefficient for the intercept  1 =coefficient for the slope Z i =1 if i th unit is in the treatment group 0 if i th unit is in the control group e i =residual for the i th unit y i =  0 +  1 Z i + e i where:

In Graph Form...

0 (Control) 1 (Treatment) ZiZi

In Graph Form... 0 (Control) 1 (Treatment) YiYi ZiZi

In Graph Form... 0 (Control) 1 (Treatment) YiYi ZiZi

In Graph Form... 0 (Control) 1 (Treatment)  0 is the intercept y-value when z=0. YiYi ZiZi

In Graph Form... 0 (Control) 1 (Treatment)  0 is the intercept y-value when z=0.  1 is the slope. YiYi ZiZi

Why Is  1 the Mean Difference? 0 (Control) 1 (Treatment)  0 is the intercept y-value when z=0.  1 is the slope. YiYi ZiZi

Why Is  1 the Mean Difference? 0 (Control) 1 (Treatment) Intuitive Explanation: Because slope is the change in y for a 1-unit change in x. YiYi ZiZi Change in y Unit change in x (i.e., z)

Why Is  1 the Mean Difference? 0 (Control) 1 (Treatment) Since the 1-unit change in x is the treatment- control difference, the slope is the difference between the posttest means of the two groups. YiYi ZiZi Change in y

Why  1 Is the Mean Difference in y i =  0 +  1 Z i + e i

Why  1 Is the Mean Difference in First, determine effect for each group: y i =  0 +  1 Z i + e i

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): y i =  0 +  1 Z i + e i

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0 e i averages to 0 across the group.

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0 y C =  0 e i averages to 0 across the group.

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): For treatment group (Z i = 1): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0 y C =  0 e i averages to 0 across the group.

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): For treatment group (Z i = 1): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0 y C =  0 y T =  0 +  1 (1) + 0 e i averages to 0 across the group.

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): For treatment group (Z i = 1): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0 y C =  0 y T =  0 +  1 (1) + 0 e i averages to 0 across the group.

Why  1 Is the Mean Difference in First, determine effect for each group: For control group (Z i = 0): For treatment group (Z i = 1): y i =  0 +  1 Z i + e i y C =  0 +  1 (0) + 0 y C =  0 y T =  0 +  1 (1) + 0 y T =  0 +  1 e i averages to 0 across the group.

Why  1 Is the Mean Difference in y i =  0 +  1 Z i + e i

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y T =  0 +  1 yTyT treatment

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y C =  0 y T =  0 +  1 y T - y C = controltreatment

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y C =  0 y T =  0 +  1 y T - y C = (  0 +  1 ) controltreatment

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y C =  0 y T =  0 +  1 y T - y C = (  0 +  1 ) -  0 controltreatment

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y C =  0 y T =  0 +  1 y T - y C = (  0 +  1 ) -  0 controltreatment y T - y C =  0 +  1 -  0

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y C =  0 y T =  0 +  1 y T - y C = (  0 +  1 ) -  0 controltreatment y T - y C =  0 +  1 -  0 

Why  1 Is the Mean Difference in Then, find the difference between the two groups: y i =  0 +  1 Z i + e i y C =  0 y T =  0 +  1 y T - y C = (  0 +  1 ) -  0 controltreatment y T - y C =  0 +  1 -  0 y T - y C =  1 

Conclusions l t-test, one-way ANOVA and regression analysis all yield same results in this case. l The regression analysis method utilizes a dummy variable for treatment. l Regression analysis is the most general model of the three.