1. STA 291 Spring 2008
Lecture 17
Dustin Lueker

2. Significance Test
A way of statistically testing a hypothesis by comparing the data to values predicted by the hypothesis
Data that fall far from the predicted values provide evidence against the hypothesis
STA 291 Spring 2008 Lecture 17

3. Logical Procedure
State a hypothesis that you would like to find evidence against
Get data and calculate a statistic
Sample mean
Sample proportion
Hypothesis determines the sampling distribution of our statistic
If the sample value is very unreasonable given our initial hypothesis, then we conclude that the hypothesis is wrong
STA 291 Spring 2008 Lecture 17

4. Elements of a Significance Test
Assumptions
Type of data, population distribution, sample size
Hypotheses
Null hypothesis H0
Alternative hypothesis H1
Test Statistic
Compares point estimate to parameter value under the null hypothesis
P-value
Uses the sampling distribution to quantify evidence against null hypothesis
Small p-value is more contradictory
Conclusion
Report p-value
Make formal rejection decision (optional)
Useful for those that are not familiar with hypothesis testing
STA 291 Spring 2008 Lecture 17

5. P-value
How unusual is the observed test statistic when the null hypothesis is assumed true?
The p-value is the probability, assuming that the null hypothesis is true, that the test statistic takes values at least as contradictory to the null hypothesis as the value actually observed
The smaller the p-value, the more strongly the data contradicts the null hypothesis
STA 291 Spring 2008 Lecture 17

6. Conclusion
In addition to reporting the p-value, sometimes a formal decision is made about rejecting or not rejecting the null hypothesis
Most studies require small p-values like p<.05 or p<.01 as significant evidence against the null hypothesis
“The results are significant at the 5% level”
α=.05
STA 291 Spring 2008 Lecture 17

7. P-values and Their Significance
Highly significant
“Overwhelming evidence”
.01<p-value<.05
Significant
“Strong evidence”
.05<p-value<.1
Not Significant
“Weak evidence
p-value>.1
“No evidence”
Whether or not a p-value is considered significant typically depends on the discipline that is conducting the study
STA 291 Spring 2008 Lecture 17

8. Terminology Significance level Alpha levelα
Number such that one rejects the null hypothesis if the p-values is less than it
Most common are .05 and .01
Needs to be chosen before analyzing the data
Why?
STA 291 Spring 2008 Lecture 17

9. Type I and Type II Errors
Decision
Reject H0
Do Not Reject H0
Condition of H0
True
Type I Error
Correct
False
Type II Error
STA 291 Spring 2008 Lecture 17

10. Type I and Type II Errors
α=probability of Type I error
β=probability of Type II error
Power=1-β
The smaller the probability of Type I error, the larger the probability of Type II error and the smaller the power
If you ask for very strong evidence to reject the null hypothesis (very small α), it is more likely that you fail to detect a real difference
In reality, α is specified, and the probability of Type II error could be calculated, but the calculations are often difficult
STA 291 Spring 2008 Lecture 17

11. Example
In a criminal trial someone is assumed innocent until proven guilty
What type of error (in terms of hypothesis testing) would be made if an innocent person is found guilty?
What type of error would be made if a guilty person is found not guilty?
What does the Power represent (1-β)?
Also, the reason we only do not reject H0 instead of saying that we accept H0 is because of the way our hypothesis tests are set up
Just like in a criminal trial someone is found not guilty instead of innocent
STA 291 Spring 2008 Lecture 17

12. How to choose α?
If the consequences of a Type I error are very serious, then α should be small
Criminal trial example
In exploratory research, often a larger probability of Type I error is acceptable
If the sample size increases, both error probabilities decrease
STA 291 Spring 2008 Lecture 17

13. How to choose α?
Which area of study would be most likely to use a very small level of significance?
Social Sciences
Medical
Physical Sciences
STA 291 Spring 2008 Lecture 17

14. Hypotheses H0: μ=μ0 H1: μ≠μ0 μ0 is the value we are testing against
Most common alternative hypothesis
This is called a two-sided hypothesis since it includes values falling on two sides of the null hypothesis (above and below)
STA 291 Spring 2008 Lecture 17

15. Test Statistic The z-score has a standard normal distribution
The z-score measures how many estimated standard errors the sample mean falls from the hypothesized population mean
The farther the sample mean falls from the larger the absolute value of the z test statistic, and the stronger the evidence against the null hypothesis
STA 291 Spring 2008 Lecture 17

16. Example
The mean age at first marriage for married men in a New England community was 22 years in 1790
For a random sample of 40 married men in that community in 1990, the sample mean age at first marriage was 26, assume the population standard deviation is 9
State the hypotheses, find the test statistic and p-value for testing whether or not the mean has changed, interpret
Make a decision, using a significance level of 5%
STA 291 Spring 2008 Lecture 17

17. P-value
Has the advantage that different test results from different tests can be compared
Always a number between 0 and 1, no matter why type of data is being examined
Probability that a standard normal distribution takes values more extreme than the observed z-score
The smaller the p-value, the stronger the evidence against the null hypothesis and in favor of the alternative hypothesis
STA 291 Spring 2008 Lecture 17

18. One-Sided Significance Tests
The research hypothesis is usually the alternative hypothesis
The alternative is the hypothesis that we want to prove by rejecting the null hypothesis
Assume that we want to prove that μ is larger than a particular number μ0
We need a one-sided test with hypotheses
Null hypothesis can also be written with an equals sign
STA 291 Spring 2008 Lecture 17

19. Example
For a large sample test of the hypothesis the z test statistic equals 1.04
Find the p-value and interpret
Suppose z=2.5 rather than 1.04, find the p-value
Does this provide stronger or weaker evidence against the null hypothesis?
Now consider the one-sided alternative
For one-sided tests, the calculation of the p-value is different
“Everything at least as extreme as the observed value” is everything above the observed value in this case
Notice the alternative hypothesis
STA 291 Spring 2008 Lecture 17

20. One-Sided vs. Two-Sided Test
Two sided tests are more common in practice
Look for formulations like
“test whether the mean has changed”
“test whether the mean has increased”
“test whether the mean is the same”
“test whether the mean has decreased”
STA 291 Spring 2008 Lecture 17

21. Example
If someone wanted to test to see if the average miles a social worker drives in a month was at least 2000 miles, what would H1 be? H0?
μ<2000
μ≤2000
μ≠2000
μ≥2000
μ>2000
μ=2000
STA 291 Spring 2008 Lecture 17