1. © LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON
STATISTICAL SIGNIFICANCE, EFFECT SIZE AND STATISTICAL POWER
© LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

2. STRUCTURE OF THE CHAPTER
Statistical significance
Concerns about statistical significance
Hypothesis testing and null hypothesis significance testing
Effect size
Statistical power
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

3. STATISTICAL SIGNIFICANCE
Statistical significance purports to test whether or not a result has been found by chance, a test of the ‘rareness’ of chance alone.
Statistical significance purports to indicate that a result is not by chance alone.
This is a test of findings being by chance alone; it is not a test of the size of an effect (e.g. a difference, a correlation).
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

4. HYPOTHESES Null hypothesis (H0) Alternative hypothesis (H1)
Significance testing assumes the null hypothesis and then requires rigorous evidence not to support it.
One commences with the null hypothesis and then tests to see if the null hypothesis is or is not supported.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

5. HYPOTHESIS TESTING Commence with a null hypothesis1
Commence with a null hypothesis 2
Set the level of significance () to be used to support or not to support the null hypothesis (the alpha () level); the alpha level is determined by the researcher 3
Compute the data 4
Determine whether the null hypothesis is supported or not supported 5
Avoid Type I and Type II errors
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

6. STATISTICAL SIGNIFICANCE
If the findings hold true 95% of the time then the statistical significance level () = 0.05
If the findings hold true 99% of the time then the statistical significance level () = 0.01
If the findings hold true 99.9% of the time then the statistical significance level () = 0.001
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

7. STATISTICAL SIGNIFICANCE AND SAMPLE SIZE
Statistical significance varies by sample size.
The smaller the sample, the larger the coefficient has to be in order to obtain statistical significance.
The larger the sample, the smaller the coefficient can be in order to obtain statistical significance.
Statistical significance can be attained either by having a large coefficient together with a small sample or having a small coefficient together with a large sample.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

8. CONCERNS ABOUT NULL HYPTOTHESIS SIGNIFICANCE TESTING
Questionable assumption of the null hypothesis as the basis for significance testing, i.e. the null hypothesis may be a false assumption.
Researchers have no way of being absolutely sure that the null hypothesis is true.
Statistical significance is a function of sample size, and it is highly likely that statistical significance will be found if large samples are used, raising the likelihood of a Type I error (a false positive).
It is highly likely that statistical significance will not be found if small samples are used, raising the likelihood of a Type II error (a false negative).
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

9. CONCERNS ABOUT NULL HYPTOTHESIS SIGNIFICANCE TESTING
Statistical significance can be attained either by having a large coefficient together with a small sample or having a small coefficient together with a large sample.
Statistical significance is arbitrary in its cut-off points (0.05, 0.01, 0.001).
Statistical significance is not the same as educational significance.
Statistical significance says nothing about what many researchers really want to know: the size of an effect. It is only a measure of chance.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

10. TYPE I AND TYPE II ERRORS
Null Hypothesis: there is no statistically significant difference between x and y.
TYPE I ERROR
The researcher rejects the null hypothesis when it is in fact true (a false positive)
 Increase significance level.
TYPE II ERROR
The researcher accepts the null hypothesis when it is in fact false (a false negative)
 Reduce significance level, increase sample size.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

11. EFFECT SIZE
Increasingly seen as preferable to statistical significance.
A way of quantifying the difference between two groups. It indicates how big the effect is, something that statistical significance does not.
For example, if one group has had an experimental treatment and the other has not (the control group), then the effect size is a measure of the effectiveness of the treatment.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

12. EFFECT SIZE It is calculated thus: Or:
Different kinds of statistical treatments use different effect-size calculations.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

13. EFFECT SIZE
Statistics for calculating effect size include r2, adjusted R2, 2, 2, Cramer’s V, Kendall’s W, Cohen’s d, Eta, Eta2. For example:
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

14. EFFECT SIZE In using Cohen’s d (a measure of difference)
0–0.20 = weak effect
0.21–0.50 = modest effect
0.51–1.00 = moderate effect
> = strong effect
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

15. EFFECT SIZE <0 +/−1 weak In measure of association/correlation:
<0 +/−3 modest
<0 +/−5 moderate
<0 +/−8 strong
 +/−0.8 very strong
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

16. EFFECT SIZE Effect sizes are susceptible to a range of influences
Restricted range: the smaller the range of scores, the greater is the possibility of a higher effect size.
Non-normal distributions: effect size usually assumes a normal distribution.
Measurement reliability: the reliability (accuracy, stability and robustness) of the instrument being used.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

17. THE POWER OF A TEST
An estimate of the ability of the test to separate the effect size from random variation.
The ability of the test to find a difference if one exists and to find no difference if none exists.
The ability of the test to avoid a Type I error (a false positive) and a Type II error (a false negative).
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

18. STATISTICAL POWER Statistical power analysis has 4 main parameters:
The effect size
The sample size (number of observations)
The alpha () significance level (usually 0.05 or lower);
The power of the statistical test (setting the acceptable  level and the desired power (1–), e.g.  of .20 and power of 0.80).
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

19.   Statistical Power TRADE-OFFS IN ,  AND POWER 1 0.5 0.5 1
Lower risk of
false negative
(Type I error) 1
0.5
The more stringent the , the less stringent the , the lower the power.
0.5 1
Low power
Statistical Power 
The less stringent the , the more stringent the , the greater the power.
Higher risk of
false negative
(Type I error)
High power 
Lower risk of false positive
(Type II error)
Higher risk of false positive
(Type II error)
Figures for  and  are significance levels

20.   THE 4 TO 1 SOLUTION Power
0.5
Set  at 0.05,  at 0.20, giving a power of 0.80
0.5 1
Power 
0.20
0.80 
0.05

21. IMPROVING STATISTICAL POWER
Use a large sample
Look for a larger effect size
Lower the  level, i.e. reduce the chance of a Type II error
Use a homogeneous sample
Use a one-tailed test
Ensure high reliability scores
Use parametric tests rather non-parametric tests (where appropriate)