Chapter 6 Making Sense of Statistical Significance: Effect Size and Statistical Power Part 1: Feb. 19, 2008

Effect Size We may reject the null and conclude there is a significant effect, but how large is it? –Effect size will estimate that; it is the amount that two populations (from our sample vs. comparison population) do not overlap Figuring effect size ( d ) Ex from last ch: –“crashed” group M=5.9, so  1 = 5.9, neutral pop  2 = 5.5,  =.8 –d = (5.9 – 5.5) /.8 =.5 for effect size  1 = experimental group (use M from sample)  2 = population or comparison group mean Population SD

Effect Size Effect size conventions – make conclusions about how large/important effect is –smallaround d =.2 (or -.2) –mediumaround d =.5 (or -.5) –largearound d =.8 (or -.8) –Ex) The difference between the ‘crashed’ and neutral groups was a significant and medium-sized effect. Speaks to ‘practical significance’ – an indication of the importance of a statistically significant effect –May have a highly statistically signif effect (due to large N), but may not have much practical significance – in reality, may not be too impressive!

Statistical Power Probability that the study will produce a statistically significant result when the research hypothesis is in fact true –That is, what is the power to correctly reject the null? –Upper right quadrant in decision table –Want to maximize our chances that our study has the power to find a true/real result Can calculate power before the study using predictions of means

Statistical Power Steps for figuring power 1. Gather the needed information: (N=16) * Mean & SD of comparison distribution (the distrib of means from Ch 5 – now known as Pop 2) * Predicted mean of experimental group (now known as Pop 1) * “Crashed” example: Pop 1 “crashed group” mean = 5.9 Pop 2 “neutral group/comparison pop” μ = 5.5,  =.8,  m = sqrt (  2 )/N  m = sqrt[(.8 2 ) / 16] =.2

Statistical Power 2. Figure the raw-score cutoff point on the comparison distribution to reject the null hypothesis (using Pop 2 info) For alpha =.05, 1-tailed test (remember we predicted the ‘crashed’ group would have higher fault ratings), z score cutoff = Convert z to a raw score (x) = z(  m ) + μ x = 1.64 (.2) = 5.83 Draw the distribution and cutoff point at 5.83, shade area to right of cutoff point  “critical/rejection region”

Statistical Power 3. Figure the Z score for this same point, but on the distribution of means for Population 1 (see ex on board) That is, convert the raw score of 5.83 to a z score using info from pop 1. –Z = (x from step 2 -  from step 1exp group )  m (from step 1 ) –(5.83 – 5.9) /.2 = -.35 –Draw another distribution & shade in everything to the right of -.35

Statistical Power 4.Use the normal curve table to figure the probability of getting a score higher than Z score from Step 3 Find % betw mean and z of -.35 (look up.35)… = 13.68% Add another 50% because we’re interested in area to right of mean too = 63.68%…that’s the power of the experiment.

Power Interpretation Our study (with N=16) has around 64% power to find a difference between the ‘crashed’ and ‘neutral’ groups if it truly exists. –Based on our estimate of what the ‘crashed’ mean will be (=5.9), so if this is incorrect, power will change. –In decision error table 1-power = beta (aka…type 2 error), so here: –Alpha =.05 (5% chance of incorrectly rejecting Null); Power =.64 (64% chance of correctly rejecting a false N); Beta =.36 (36% chance of incorrectly failing to reject N)