1. Power

2. The Four Components to a Statistical Conclusion The number of units (e.g., people) accessible to study The salience of the program relative to the noise The odds the observed result is due to chance The odds you’ll observe a treatment effect when it occurs Sample size Effect size Alpha level Power

3. The Effect Size Is a ratio of... Signal Noise

4. The Effect Size Is a ratio of... Treatment or program group, Compared to the control group With the standard error of The difference factored in Signal Noise

5. What Is the Decision Matrix? A basic decision diagram showing the steps in involved in a statistical conclusion A basic decision diagram showing the steps in involved in a statistical conclusion The central decision involves accepting the null or alternative hypothesis The central decision involves accepting the null or alternative hypothesis

6. The Decision Matrix In reality What we conclude

7. The Decision Matrix In reality What we conclude Null true Alternative false In reality... There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong

8. The Decision Matrix In reality What we conclude Null true Alternative false In reality... Accept null Reject alternative We say... There is no real program effect There is no difference, gain Our theory is wrong There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong

9. The Decision Matrix In reality What we conclude Null true Alternative false In reality... Accept null Reject alternative We say... There is no real program effect There is no difference, gain Our theory is wrong There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong 1-  THE CONFIDENCE LEVEL The odds of saying there is no effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is none

10. The Decision Matrix In reality What we conclude Null true Alternative false In reality... Reject null Accept alternative We say... There is a real program effect There is a difference, gain Our theory is correct There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong

11. The Decision Matrix In reality What we conclude Null true Alternative false In reality... Reject null Accept alternative We say... There is a real program effect There is a difference, gain Our theory is correct There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong  TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is one

12. The Decision Matrix In reality What we conclude Null false Alternative true In reality... There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct

13. The Decision Matrix In reality What we conclude Null false Alternative true In reality... Accept null Reject alternative We say... There is no real program effect There is no difference, gain Our theory is wrong There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct

14. The Decision Matrix In reality What we conclude Null false Alternative true In reality... Accept null Reject alternative We say... There is no real program effect There is no difference, gain Our theory is wrong There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct TYPE II ERROR  The odds of saying there is no effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is none

15. The Decision Matrix In reality What we conclude Null false Alternative true In reality... Reject null Accept alternative We say... There is a real program effect There is a difference, gain Our theory is correct There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct

16. The Decision Matrix In reality What we conclude Null false Alternative true In reality... Reject null Accept alternative We say... There is a real program effect There is a difference, gain Our theory is correct There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct 1-  POWER The odds of saying there is an effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is one

17. The Decision Matrix In reality What we conclude Null true Null false Alternative false Alternative true In reality... Accept null Reject alternative Reject null Accept alternative We say... There is no real program effect There is no difference, gain Our theory is wrong We say... There is a real program effect There is a difference, gain Our theory is correct There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct 1-  THE CONFIDENCE LEVEL TYPE II ERROR  The odds of saying there is no effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is none The odds of saying there is no effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is none  1-  TYPE I ERROR POWER The odds of saying there is an effect or gain when in fact there is none The odds of saying there is an effect or gain when in fact there is one # of times out of 100 when there is no effect, we’ll say there is one # of times out of 100 when there is an effect, we’ll say there is one

18. The Decision Matrix In reality What we conclude Null true Null false Alternative false Alternative true In reality... Accept null Reject alternative Reject null Accept alternative We say... There is no real program effect There is no difference, gain Our theory is wrong We say... There is a real program effect There is a difference, gain Our theory is correct There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct 1-  THE CONFIDENCE LEVEL TYPE II ERROR   1-  TYPE I ERROR POWER

19. The Decision Matrix In reality What we conclude Null true Null false Alternative false Alternative true In reality... Accept null Reject alternative Reject null Accept alternative We say... There is no real program effect There is no difference, gain Our theory is wrong We say... There is a real program effect There is a difference, gain Our theory is correct There is no real program effectThere is no real program effect There is no difference, gainThere is no difference, gain Our theory is wrongOur theory is wrong There is a real program effectThere is a real program effect There is a difference, gainThere is a difference, gain Our theory is correctOur theory is correct 1-  THE CONFIDENCE LEVEL TYPE II ERROR   1-  TYPE I ERROR POWER CORRECT CORRECT