1. Hypothesis test flow chart
START HERE
number of
correlations
number of
variables
χ2 test for
independence
(19.9) Table I
Test H0: r=0
(17.2)
Table G 1
correlation (r)
frequency
data 2
Measurement
scale 2
Means 1
basic χ2 test
(19.5)
Table I
Test H0: r1= r2
(17.4)
Tables H and A
Do you know s?
number of
means
number of
factors
z -test
(13.1)
Table A
Yes 1
More than 2 2
2-way ANOVA
Ch 21
Table E No 2 1
independent
samples?
1-way ANOVA
Ch 20
Table E
t -test
(13.14)
Table D
Yes No
Test H0: m1= m2
(15.6)
Table D
Test H0: D=0
(16.4)
Table D

2. Chapter 13: Interpreting the Results of Hypothesis Testing
‘statistically significant’ does not mean ‘important’
IQ’s of UW undergraduates
Suppose we measured the IQ’s of 10,000 UW undergraduates and found a mean IQ of If we were to conduct a one-tailed z-test to determine if this mean is greater than the US population that has a mean of 100 and a standard deviation of 15. -3 -2 -1 1 2 3 z
We’d find that we could reject H0 with a=.05.
But is a difference of 0.3 IQ points important?
area = a = .05
z=2

3. If you want to read a lot about statistically significant results that may or may not be important…

4. Some journals require the authors to report the ‘effect size’, along with the outcomes of statistical tests to let the reader interpret whether the effect is ‘big’ enough to be important.
Remember, to calculate t, we divide by the standard error of the mean:
But the standard error of the mean shrinks with increasing n.
We need a measure of the size of the difference between our observation and the null hypothesis that doesn’t depend on experimental parameters like n.

5. Effect size: the difference between our observation and the null hypothesis in terms of standard deviations. Formally: effect size is “an estimate of the degree to which the treatment effect is present in the population, expressed as a number free of the original measurement unit”.
One example of effect size is Cohen’s d:
Where mhyp is the mean for the null hypothesis. This is just like converting the sample mean to a z score.
A more common example is Hedge’s g, which is used when we don’t know the standard deviation of the population. It’s our best estimate of Cohen’s d:
This is just like calculating a value for the t-distribution except we divide by sX instead of the standard error of the mean 𝑠 𝑥

6. Back to our made-up IQ example where we had a mean of 100
Back to our made-up IQ example where we had a mean of and a standard deviation of 15
The effect size is:
The study found that UW IQ’s are only 0.02 standard deviations above 100.
This is a small effect size, even though it is statistically significant.

7. 0.8 0.8 is large, 0.5 is medium 0.2 is small. 0.5
Reporting effect size has the advantage that since it doesn’t depend on n, the value is more easily compared across studies.
0.8
0.5
0.2
A conventional interpretation of effect size is that (in absolute value):
0.8 is large,
0.5 is medium
0.2 is small.

8. Example: What is the effect size for the ‘Freshman 15’ example (for women)?
Male Freshman
Female Freshman
Mean
3.1 pounds
3.5 sd
10.1
10.3 n
2536
2151
This would be considered large effect size.

9. Example: What about comparing this result to the null hypothesis that there was no weight gain the first year (H0: mx =0) ?
Male Freshman
Female Freshman
Mean
3.1 pounds
3.5 sd
10.1
10.3 n
2536
2151
This would be considered small to medium effect size – even though it is highly statistically significant.
(t = 16.76, you can do the t-test yourself as an exercise).

10. There are two unavoidable types of errors in hypothesis testing: type I and type II errors.
True state of the world
HO is true
HO is false
Correctly fail to
reject HO (1-a)
Type II Error
(b)
Fail to reject HO
Decision based on your sample
Type I Error
(a)
Correctly reject H0
(1-b = power)
Reject HO
A Type I error is when we reject H0 when it is actually true. Pr(Type I error) = a
A Type II error is when we fail to reject H0 even though it false. Pr(Type II error) = b
More commonly, we talk about the probability of correctly rejecting H0,
The probability of this happening is called power:
Power = Pr(correct rejection of HO) = 1-b.

11. True state of the world
HO is true
HO is false
Correctly fail to
reject HO (1-a)
Type II Error
(b)
Fail to reject HO
Decision based on your sample
Type I Error
(a)
Correctly reject H0
(1-b = power)
Reject HO

12. Alpha (a) is therefore the probability that a Type I error will occur.
Type I errors (a)
A Type I error occurs when our statistic (z or t) falls within the region or rejection even though the null hypothesis is true.
For example, for a one-tailed z-test using a = .05, the distribution of z scores and the rejection regions look like this: -4 -3 -2 -1 1 2 3 4
z score
Pr(Type I error) = a
Alpha (a) is therefore the probability that a Type I error will occur.

13. Type II errors
At Type II error happens when the null hypothesis is false but you fail to reject it anyway.
To calculate the probability of a type II error, we need to know the true distribution of the population.
This is weird because the true distribution of the population is the thing we’re trying to figure out in the first place.

14. Type II errors: beta (b) and power (1-b)
Type II errors happen only if the null hypothesis is false.
For example, suppose we’re conducting a one-tailed z-test with a = .05, and the true population mean has a mean z score of 1 (mtrue = 1). We still use the same critical value that we did under the null hypothesis. But now the distribution of z-values is centered around z=1.
mtrue = 1
Zcrit = 1.645
mhyp = 0
1-b = power (blue shaded area)
a = Pr(type I error) (red shaded area) -3 -2 -1 1 2 3 4 5
z-score
The blue shaded region is the probability of correctly rejecting the null hypothesis.
Type II errors happen when z falls outside the rejection region, so the probability of making a Type II error is 1- blue shaded area.

15. Type II errors: beta (b) mtrue = 1 Zcrit = 1.645
mhyp = 0
1-b = power (blue shaded area)
a (red shaded area) -3 -2 -1 1 2 3 4 5
z-score
Calculating power, the probability of correctly rejecting HO
1) Find the rejection region under the null hypothesis:
With a = .05, zcrit = (Table A, column C), so the rejection region is z>1.645
2) The new rejection region will by shifted down by utrue – uhyp = 1
= 0.645, so the new rejection region is z>0.645
3) Find the area in the new rejection region
The power is the area for z above is (Table A, Column C)

16. 1-b = power (blue shaded area)
mtrue = 1
Zcrit = 1.645
mhyp = 0
1-b = power (blue shaded area)
a (red shaded area) -3 -2 -1 1 2 3 4 5
z-score
Power is the probability of correctly rejecting the null hypothesis, which is the area in the rejection region.
Power in this example is: Pr(z>0.645) = 1-b = .2611
More power is good. Power is the probability of correctly finding an effect in your experiment.
A ‘desirable’ level of power is .8

17. Example: Between 1930 and 1980 the year-to-year average temperature in the Northern Hemisphere varied with a standard deviation of degrees. Consider this to be the population standard deviation for temperatures.
Suppose we wanted to calculate the mean of the temperatures over a random sample of 12 years. What is the standard error of this mean?
-0.2
-0.1
0.1
0.2
Temperature (deg)

18. What temperature above the mean exceeds 99% of all temperatures?
Example: Between 1930 and 1980 the year-to-year average temperature in the Northern Hemisphere varied with a standard deviation of degrees. Consider this to be the population standard deviation for temperatures. (These values were taken from the NOAA website)
What temperature above the mean exceeds 99% of all temperatures?
-0.2
-0.1
0.1
0.2
Temperature (deg)
z = .1668
From Table A, Pr(z>2.33) = .01
x = (2.33)(.0716) = degrees

19. x = (2.33)(.0716) = degrees
If we were to calculate the temperature averaged over the 12 years from 2000 to 2011, there is a 1% chance that we’d find an increase of degrees or more from 1930 to 1980.
In other words, if null hypothesis is true, that the temperature has not increased since the period 1930 to 1980, we’d make a Type I error if we found an average temperature increase of degrees or more. The probability of this type I error is 1%
-0.2
-0.1
0.1
0.2
Temperature (deg)
z = .1668

20. Pr(x<.1668) = Pr(z< (.1668-.3)/.0716)) = Pr(z<-1.86) = .0314
x = (2.33)(.0716) = degrees
Now suppose that you actually measured that the average temperature over the years 2000 to 2011 has increased by 0.3 degrees above the average from 1930 to 1980.
If we assume an alpha value of .01, what is the probability that this is a type II error?
In other words, given a new normal distribution with standard deviation of and a mean of 0.3, what is the probability that a sample will be less than the old critical value of .1668?
0.1
0.2
0.3
0.4
0.5
Temperature (deg)
z = .1668
Pr(x<.1668) = Pr(z< ( )/.0716))
= Pr(z<-1.86) =

21. Pr(correct rejection of Ho) is 1-Pr(Type II error) = 1-.0314 = .9686
That is, if the real temperature increase is 0.3 degrees, we would fail to detect it 3.14% of the time.
If the probability of a type II error is.0314, then what is the probability that we will correctly detect a temperature change of 0.3 degrees or higher?
Pr(correct rejection of Ho) is 1-Pr(Type II error) = = .9686
The probability of a correct rejection of Ho is the power of the test. The power here is We will correctly reject Ho 96.86% of the time.
z = .1668
0.1
0.2
0.3
0.4
0.5
Temperature (deg)

22. Not quite impossible, but effectively so.
We just showed that we have a 96.86% chance of correctly detecting a 0.3 average temperature increase since 2000.
The actual measured temperature increase has been 1.67 degrees since
The probability of this happening under the null hypothesis is Pr(z < -21).
Not quite impossible, but effectively so.
1930
1940
1950
1960
1970
1980
1990
2000
2010 -1
-0.5
0.5 1
1.5 2
2.5
Years
Relative temperature (deg)

23. Example: IQs are normally distributed with a mean of 100 and a standard deviation of 15. Suppose you sampled 100 students and calculated a sample mean and are about to test for a significant increase in IQ using a one-tailed z-test using a=.05. What is the power of this test under the assumption that the true population mean for the group that we’re sampling is 103?
Answer: First, we’ll convert everything to z-scores. This makes mhyp = 0 (always), and

24. 1) Find the critical value of t under null hypothesis:
To calculate power:
1) Find the critical value of t under null hypothesis:
With a = .05, zcrit = 1.64 (Table A, column C), so the rejection region is z > 1.64
2) The new rejection region will by shifted over by utrue – uhyp = 2-0 = 2
z > , which is z `> -.36
3) Find 1-b, the area in the new rejection region
Pr(z > -.36) = = .6406
A power of means that there is a 64.06% chance
of correctly rejecting the null hypothesis
(or not making a type II error).
power = 1-b = .6406 -4 -3 -2 -1 1 2 3 4
z-score

25. Things that affect power: Variability of the measure
mtrue = 1.0
Power increases as the standard error of the mean decreases.
Ways to decrease the standard error of the mean:
Increase the sample size
Make more accurate measurements

26. Things that affect power: level of significance (a) a=? mtrue = 1.0
Power decreases as alpha (a) decreases. -3 -2 -1 1 2 3 4 5
z score
power =0.2595 -3 -2 -1 1 2 3 4 5
z score
power =0.1685
a=.05
a=.025 -3 -2 -1 1 2 3 4 5
z score
power =0.0924 -3 -2 -1 1 2 3 4 5
z score
power =0.0183
a=.01
a=.001
This is a classic tradeoff: The less willing we are to make a Type I error, the more likely we are going to make a Type II error.

27. mtrue is the one thing we don’t know (but want to estimate).
Things that affect power: difference between utrue and uhyp
a=.05
mtrue = ?
Power increases with effect size: as the difference between means for the true population and the null hypothesis increases. -3 -2 -1 1 2 3 4 5
z score
power =0.0815 -3 -2 -1 1 2 3 4 5
z score
power =0.1261
mtrue = 0.25
mtrue = 0.5 -3 -2 -1 1 2 3 4 5
z score
power =0.2595 -3 -2 -1 1 2 3 4 5 6 7 8
z score
power =0.9907
mtrue = 4.0
mtrue = 1.0
We don’t have control over this:
mtrue is the one thing we don’t know (but want to estimate).

28. Power curve: shows how power increases with effect size Two-tail a=.051
Sample
size = 50
0.9
0.8
0.7
0.6
Power
0.5
0.4
0.3
0.2
0.1
0.2
0.4
0.6
0.8 1
Effect size (d)

29. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.01, 1-tail, 1 mean

30. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.05, 1-tail, 1 mean

31. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.01, 2-tails, 1 mean

32. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.05, 2-tails, 1 mean

33. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.01, 1-tail, 2 means

34. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.05, 1-tail, 2 means

35. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.01, 2-tails, 2 means

36. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.05, 2-tails, 2 means

37. Example: Suppose we’re conducting a two-tailed t-test with one mean with a = .05 with a sample size of n=50. How much of an effect size do we need to obtain a power value of 0.8?
Answer: Looking at the appropriate family of power curves, the curve with n=50 passes through a power value of 0.8 when the effect size is 0.4.
Example: Suppose we’re conducting a one-tailed t-test with one mean with a = .01 and we have an effect size of How large of a sample size do we need to get a power value of 0.8?
Answer: Looking at the appropriate family of power curves, looking at a power value of 0.4, the curve with n=30 passes through a power value of 0.8.

38. Example: You decide to sample the test scores of 63 dazzling cats from a population and obtain a mean test scores of 25.6 and a standard deviation of Using an alpha value of α = 0.01, is this observed mean significantly different than an expected test scores of 25?
What is the effect size? What is the power?

39. Example: You decide to sample the test scores of 63 dazzling cats from a population and obtain a mean test scores of 25.6 and a standard deviation of Using an alpha value of α = 0.01, is this observed mean significantly different than an expected test scores of 25?
What is the effect size? What is the power?
Answer: (Two tailed t-test for one mean) We fail to reject H0 (t(62) = 1.72, tcrit = ±2.6575). The test scores of dazzling cats is not significantly different than 25. Effect size: Power =

40. Power curves for independent samples t-test (2 means):
Remember, the power of a test is the probability of correctly rejecting H0 when it is false.
Power depends on a, nX, nY, and the effect size.

41. Example (again): The heights of the 45 students in Psych 315 with fathers above 70 inches has a mean of 66.8 inches and a standard deviation of 4.14 inches. The heights of the remaining 51 students has a mean of 64.9 and a standard deviation of 3.62 inches. What is the effect size? (use a = .05, two tailed)
This is a medium effect size. Remember it was a ‘significant’ t-test.
Assuming that our observed mean is the true population mean, what is the power of this test?
How large of a sample would we need to obtain an effect size of 0.8?

42. 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1
1.1
1.2
1.3
1.4
n=8 10 12 15 20 25 30 40 50 75
100
150
250
500
1000
Effect size (d)
Power a
= 0.05, 2-tails, 2 means
For an effect size of 0.49 and sample sizes of about 50, the power is about We’d need between 50 and 75 subjects per group to get an effect size of 0.8.

43. Example: Suppose we’re conducting a two-tailed t-test with a =
Example: Suppose we’re conducting a two-tailed t-test with a = .05 with a sample size of n=50. How much of an effect size do we need to obtain a power value of 0.8?
Answer: Looking at the appropriate family of power curves, the curve with n=50 passes through a power value of 0.8 when the effect size is 0.4.
Example: Suppose we’re conducting a one-tailed t-test with a = .01 and we have an effect size of How large of a sample size do we need to get a power value of 0.8?
Answer: Looking at the appropriate family of power curves, looking at a power value of 0.4, the curve with n=30 passes through a power value of 0.8.

44. Example: You decide to sample the test scores of 63 dazzling cats from a population and obtain a mean test scores of 25.6 and a standard deviation of Using an alpha value of α = 0.01, is this observed mean significantly different than an expected test scores of 25?
What is the effect size? What is the power?

45. Example) You decide to sample the test scores of 63 dazzling cats from a population and obtain a mean test scores of 25.6 and a standard deviation of Using an alpha value of α = 0.01, is this observed mean significantly different than an expected test scores of 25?
What is the effect size? What is the power?
Answer) (Two tailed t-test for one mean) We fail to reject H0 (t(62) = 1.72, tcrit = ±2.6575). The test scores of dazzling cats is not significantly different than 25. Effect size: Power =