Section 9.1 First Day The idea of a significance test What is a p-value?

Inference There are basically two main types of inference. Confidence Intervals Significance Tests a.k.a. Hypothesis Tests Used to estimate the value of a parameter Used to test a claim

The logic behind significance tests I’m a really, really good trashketball shooter. I claim I make at least 80% of my shots. You say “prove it!” and challenge me to a 50-shot trial. If I make 39 out of 50 shots, would you call me a liar? What if I make only 32 out of 50 shots? So, the logic is this. You could compute how likely it is (i.e. the probability) that I only make 32 out of 50 shots if I’m truly an 80% free throw shooter. The lower that probability is, the more evidence you have to reject my claim and say that you think I’m lying. The farther away the sample statistic is away from the claimed value of the parameter, the more evidence you have against the claim.

The Reasoning of Significance Tests Based on the evidence, we might conclude my claim is incorrect.In reality, there are two possible explanations for the fact that I made only 64% of my free throws. 1) The player’s claim is correct (p = 0.8), and by bad luck, a very unlikely outcome occurred. 2) The population proportion is actually less than 0.8, so the sample result is not an unlikely outcome. An outcome that would rarely happen if a claim were true is good evidence that the claim is not true. Basic Idea

Stating the hypotheses The first step in a test of significance is to state the hypotheses. The first claim is called the null hypothesis. It is abbreviated H 0. I won’t call it a Ho, but you will. “Null” literally means nothing. So, the null hypothesis stands for “no difference” or “no effect.” When we’re testing whether a medicine works, our null hypothesis will be that it doesn’t work – there is no effect from the drug. The second claim is called the alternative hypothesis. It is abbreviated H a. It is the claim we hope or suspect to be true. In the free-throw shooter example, our hypotheses are H 0 : p = 0.80 H a : p < 0.80 where p is the long-run proportion of made free throws.

General format In any significance test, the null hypothesis has the form H 0 : parameter = value The alternative hypothesis has one of the forms H a : parameter < value H a : parameter > value H a : parameter ≠ value To determine the correct form of H a, read the problem carefully. “One-sided” alternatives “Two-sided” alternative NEVER WRITE A HYPOTHESIS ABOUT A STATISTIC!!!

Interpreting P -Values The null hypothesis H 0 states the claim that we are seeking evidence against. The probability that measures the strength of the evidence against a nullhypothesis is called a P -value. Definition: The probability, computed assuming H 0 is true, that the statistic would take a value as extreme as or more extreme than the one actually observed is called the P-value of the test. The smaller the P-value, the stronger the evidence against H 0 provided by the data. Small P-values are evidence against H 0 because they say that the observed result is unlikely to occur when H 0 is true. Large P-values fail to give convincing evidence against H 0 because they say that the observed result is likely to occur by chance when H 0 is true.

Example: Studying Job Satisfaction a)Explain what it means for the null hypothesis to be true in this setting. b) Interpret the P-value in context. In this setting, H 0 : µ = 0 says that the mean difference in satisfaction scores (self-paced - machine-paced) for the entire population of assembly-line workers at the company is 0. If H 0 is true, then the workers don’t favor one work environment over the other, on average. For the job satisfaction study, the hypotheses are H 0 : µ = 0 H a : µ ≠ 0 The P-value is the probability of observing a sample result as extreme or more extreme in the direction specified by H a just by chance when H 0 is actually true. An outcome that would occur so often just by chance (almost 1 in every 4 random samples of 18 workers) when H 0 is true is not convincing evidence against H 0. We fail to reject H 0 : µ = 0. An outcome that would occur so often just by chance (almost 1 in every 4 random samples of 18 workers) when H 0 is true is not convincing evidence against H 0. We fail to reject H 0 : µ = 0.

Statistical Significance The final step in performing a significance test is to draw a conclusion about the competing claims you were testing. We will make one of two decisions basedon the strength of the evidence against the null hypothesis (and in favor of thealternative hypothesis) -- reject H 0 or fail to reject H 0. If our sample result is too unlikely to have happened by chance assuming H 0 is true, then we’ll reject H 0. Otherwise, we will fail to reject H 0. Note: A fail-to-reject H 0 decision in a significance test doesn’t mean that H 0 is true. For that reason, you should never “accept H 0 ” or use language implying that you believe H 0 is true. In a nutshell, our conclusion in a significance test comes down to P-value LOW → reject H 0 → conclude H a (in context) P-value HIGH → fail to reject H 0 → cannot conclude H a (in context)