Statistics: Unlocking the Power of Data Lock 5 Hypothesis Testing: Significance STAT 101 Dr. Kari Lock Morgan 9/27/12 SECTION 4.3 Significance level Statistical conclusions Type I and II errors

Statistics: Unlocking the Power of Data Lock 5 Office Hours My office hours next week will be Wednesday 1 – 3pm, NOT Monday (and Thurs 1 – 2:30 as always)

Statistics: Unlocking the Power of Data Lock 5 Randomization Distributions p-values can be calculated by randomization distributions:  simulate samples, assuming H 0 is true  calculate the statistic of interest for each sample  find the p-value as the proportion of simulated statistics as extreme as the observed statistic Let’s do a randomization distribution for a randomized experiment…

Statistics: Unlocking the Power of Data Lock 5 In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug), and then followed to see who relapsed Question of interest: Is Desipramine better than Lithium at treating cocaine addiction? Cocaine Addiction

Statistics: Unlocking the Power of Data Lock 5 What are the null and alternative hypotheses? What are the possible conclusions? Cocaine Addiction Reject H 0 ; Desipramine is better than Lithium Do not reject H 0 : We cannot determine from these data whether Desipramine is better than Lithium

Statistics: Unlocking the Power of Data Lock 5 RRRRRR RRRRRR RRRRRR RRRRRR RRRRRR RRRRRR RRRRRR RRRRRR RRRR RRRRRR RRRRRR RRRRRR RRRR RRRRRR RRRRRR RRRRRR Desipramine Lithium 1. Randomly assign units to treatment groups

Statistics: Unlocking the Power of Data Lock 5 RRRR RRRRRR RRRRRR NNNNNN RRRRRR RRRRNN NNNNNN RR NNNNNN R = Relapse N = No Relapse RRRR RRRRRR RRRRRR NNNNNN RRRRRR RRRRRR RRNNNN RR NNNNNN 2. Conduct experiment 3. Observe relapse counts in each group Lithium Desipramine 10 relapse, 14 no relapse18 relapse, 6 no relapse 1. Randomly assign units to treatment groups

Statistics: Unlocking the Power of Data Lock 5 To see if a statistic provides evidence against H 0, we need to see what kind of sample statistics we would observe, just by random chance, if H 0 were true Measuring Evidence against H 0

Statistics: Unlocking the Power of Data Lock 5 “by random chance” means by the random assignment to the two treatment groups “if H 0 were true” means if the two drugs were equally effective at preventing relapses (equivalently: whether a person relapses or not does not depend on which drug is taken) Simulate what would happen just by random chance, if H 0 were true… Cocaine Addiction

Statistics: Unlocking the Power of Data Lock 5 RRRR RRRRRR RRRRRR NNNNNN RR RRRR RRRRNN NNNNNN RR NNNNNN 10 relapse, 14 no relapse18 relapse, 6 no relapse

Statistics: Unlocking the Power of Data Lock 5 RRRRRR RRRRNN NNNNNN NNNNNN RRRRRR RRRRRR RRRRRR NNNNNN RNRN RRRRRR RNRRRN RNNNRR NNNR NRRNNN NRNRRN RNRRRR Simulate another randomization Desipramine Lithium 16 relapse, 8 no relapse12 relapse, 12 no relapse

Statistics: Unlocking the Power of Data Lock 5 RRRR RRRRRR RRRRRR NNNNNN RR RRRR RNRRNN RRNRNR RR RNRNRR Simulate another randomization Desipramine Lithium 17 relapse, 7 no relapse11 relapse, 13 no relapse

Statistics: Unlocking the Power of Data Lock 5 Simulate Your Own Sample In the experiment, 28 people relapsed and 20 people did not relapse. Create cards or slips of paper with 28 “R” values and 20 “N” values. Pool these response values together, and randomly divide them into two groups (representing Desipramine and Lithium) Calculate your difference in proportions Plot your statistic on the class dotplot To create an entire randomization distribution, we simulate this process many more times with technology: StatKeyStatKey

Statistics: Unlocking the Power of Data Lock 5 p-value

Statistics: Unlocking the Power of Data Lock 5 Formal Decisions If the p-value is small:  REJECT H 0  the sample would be extreme if H 0 were true  the results are statistically significant  we have evidence for H a If the p-value is not small:  DO NOT REJECT H 0  the sample would not be too extreme if H 0 were true  the results are not statistically significant  the test is inconclusive; either H 0 or H a may be true

Statistics: Unlocking the Power of Data Lock 5 A formal hypothesis test has only two possible conclusions: 1.The p-value is small: reject the null hypothesis in favor of the alternative 2.The p-value is not small: do not reject the null hypothesis Formal Decisions How small?

Statistics: Unlocking the Power of Data Lock 5 Significance Level The significance level, , is the threshold below which the p-value is deemed small enough to reject the null hypothesis p-value <   Reject H 0 p-value >   Do not Reject H 0

Statistics: Unlocking the Power of Data Lock 5 Significance Level If the p-value is less than , the results are statistically significant, and we reject the null hypothesis in favor of the alternative If the p-value is not less than , the results are not statistically significant, and our test is inconclusive Often  = 0.05 by default, unless otherwise specified

Statistics: Unlocking the Power of Data Lock 5 Resveratrol, an ingredient in red wine and grapes, has been shown to promote weight loss in rodents, and has recently been investigated in primates (specifically, the Grey Mouse Lemur). A sample of lemurs had various measurements taken before and after receiving resveratrol supplementation for 4 weeks Red Wine and Weight Loss BioMed Central (2010, June 22). “Lemurs lose weight with ‘life-extending’ supplement resveratrol. Science Daily.

Statistics: Unlocking the Power of Data Lock 5 Red Wine and Weight Loss In the test to see if the mean resting metabolic rate is higher after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) a) Yes b) No The p-value is lower than  = 0.05, so the results are statistically significant and we reject H 0.

Statistics: Unlocking the Power of Data Lock 5 Red Wine and Weight Loss In the test to see if the mean body mass is lower after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) a) Yes b) No The p-value is lower than  = 0.05, so the results are statistically significant and we reject H 0.

Statistics: Unlocking the Power of Data Lock 5 Red Wine and Weight Loss In the test to see if locomotor activity changes after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) a) Yes b) No The p-value is not lower than  = 0.05, so the results are not statistically significant and we do not reject H 0.

Statistics: Unlocking the Power of Data Lock 5 Red Wine and Weight Loss In the test to see if mean food intake changes after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) a) Yes b) No The p-value is lower than  = 0.05, so the results are statistically significant and we reject H 0.

Statistics: Unlocking the Power of Data Lock 5 H 0 : X is an elephant H a : X is not an elephant Would you conclude, if you get the following data? X walks on two legs X has four legs Elephant Example Reject H 0 ; evidence that X is not an elephant Although we can never be certain! Do not reject H 0 ; we do not have sufficient evidence to determine whether X is an elephant

Statistics: Unlocking the Power of Data Lock 5 “For the logical fallacy of believing that a hypothesis has been proved to be true, merely because it is not contradicted by the available facts, has no more right to insinuate itself in statistical than in other kinds of scientific reasoning…” -Sir R. A. Fisher Never Accept H 0 “Do not reject H 0 ” is not the same as “accept H 0 ”! Lack of evidence against H 0 is NOT the same as evidence for H 0 !

Statistics: Unlocking the Power of Data Lock 5 Statistical Conclusions In a hypothesis test of H 0 :  = 10 vs H a :  < 10 the p-value is With α = 0.05, we conclude: a) Reject H 0 b) Do not reject H 0 c) Reject H a d) Do not reject H a The p-value of is less than α = 0.05, so we reject H 0

Statistics: Unlocking the Power of Data Lock 5 Statistical Conclusions In a hypothesis test of H 0 :  = 10 vs H a :  < 10 the p-value is With α = 0.01, we conclude: a) There is evidence that  = 10 b) There is evidence that  < 10 c) We have insufficient evidence to conclude anything

Statistics: Unlocking the Power of Data Lock 5 Statistical Conclusions In a hypothesis test of H 0 :  = 10 vs H a :  < 10 the p-value is With α = 0.01, we conclude: a) Reject H 0 b) Do not reject H 0 c) Reject H a d) Do not reject H a The p-value of 0.21 is not less than α = 0.01, so we do not reject H 0

Statistics: Unlocking the Power of Data Lock 5 Statistical Conclusions In a hypothesis test of H 0 :  = 10 vs H a :  < 10 the p-value is With α = 0.01, we conclude: a) There is evidence that  = 10 b) There is evidence that  < 10 c) We have insufficient evidence to conclude anything

Statistics: Unlocking the Power of Data Lock 5 Informal strength of evidence against H 0 : Formal decision of hypothesis test, based on  = 0.05 : Statistical Conclusions

Statistics: Unlocking the Power of Data Lock 5 Multiple Sclerosis and Sunlight It is believed that sunlight offers some protection against multiple sclerosis, but the reason is unknown Researchers randomly assigned mice to one of: Control (nothing) Vitamin D Supplements UV Light All mice were injected with proteins known to induce a mouse form of MS, and they observed which mice got MS Seppa, Nathan. “Sunlight may cut MS risk by itself”, Science News, April 24, 2010 pg 9, reporting on a study appearing March 22, 2010 in the Proceedings of the National Academy of Science.

Statistics: Unlocking the Power of Data Lock 5 Multiple Sclerosis and Sunlight For each situation below, write down  Null and alternative hypotheses  Informal description of the strength of evidence against H 0  Formal decision about H 0, using α = 0.05  Conclusion in the context of the question In testing whether UV light provides protection against MS (UV light vs control group), the p-value is In testing whether Vitamin D provides protection against MS (Vitamin D vs control group), the p- value is 0.47.

Statistics: Unlocking the Power of Data Lock 5 Multiple Sclerosis and Sunlight In testing whether UV light provides protection against MS (UV light vs control group), the p-value is H 0 : p UV – p C = 0 H a : p UV – p C < 0 We have strong evidence against H 0 Reject H 0 We have strong evidence that UV light provides protection against MS, at least in mice.

Statistics: Unlocking the Power of Data Lock 5 Multiple Sclerosis and Sunlight In testing whether Vitamin D provides protection against MS (Vitamin D vs control group), the p-value is H 0 : p D – p C = 0 H a : p D – p C < 0 We have little evidence against H 0 Do not reject H 0 We cannot conclude anything about Vitamin D and MS.

Statistics: Unlocking the Power of Data Lock 5 There are four possibilities: Errors Reject H 0 Do not reject H 0 H 0 true H 0 false TYPE I ERROR TYPE II ERROR Truth Decision A Type I Error is rejecting a true null A Type II Error is not rejecting a false null

Statistics: Unlocking the Power of Data Lock 5 In the test to see if resveratrol is associated with food intake, the p-value is o If resveratrol is not associated with food intake, a Type I Error would have been made In the test to see if resveratrol is associated with locomotor activity, the p-value is o If resveratrol is associated with locomotor activity, a Type II Error would have been made Red Wine and Weight Loss

Statistics: Unlocking the Power of Data Lock 5 A person is innocent until proven guilty. Evidence must be beyond the shadow of a doubt. Types of mistakes in a verdict? Convict an innocent Release a guilty HoHo HaHa  Type I error Type II error Analogy to Law p-value from data

Statistics: Unlocking the Power of Data Lock 5 The probability of making a Type I error (rejecting a true null) is the significance level, α α should be chosen depending how bad it is to make a Type I error Probability of Type I Error

Statistics: Unlocking the Power of Data Lock 5 If the null hypothesis is true: 5% of statistics will be in the most extreme 5% 5% of statistics will give p-values less than % of statistics will lead to rejecting H 0 at α = 0.05 If α = 0.05, there is a 5% chance of a Type I error Distribution of statistics, assuming H 0 true: Probability of Type I Error

Statistics: Unlocking the Power of Data Lock 5 If the null hypothesis is true: 1% of statistics will be in the most extreme 1% 1% of statistics will give p-values less than % of statistics will lead to rejecting H 0 at α = 0.01 If α = 0.01, there is a 1% chance of a Type I error Distribution of statistics, assuming H 0 true: Probability of Type I Error

Statistics: Unlocking the Power of Data Lock 5 Probability of Type II Error The probability of making a Type II Error (not rejecting a false null) depends on  Effect size (how far the truth is from the null)  Sample size  Variability  Significance level

Statistics: Unlocking the Power of Data Lock 5 Choosing α By default, usually α = 0.05 If a Type I error (rejecting a true null) is much worse than a Type II error, we may choose a smaller α, like α = 0.01 If a Type II error (not rejecting a false null) is much worse than a Type I error, we may choose a larger α, like α = 0.10

Statistics: Unlocking the Power of Data Lock 5 Come up with a hypothesis testing situation in which you may want to… Use a smaller significance level, like  = 0.01 Use a larger significance level, like  = 0.10 Significance Level

Statistics: Unlocking the Power of Data Lock 5 Results are statistically significant if the p-value is less than the significance level, α In making formal decisions, reject H 0 if the p- value is less than α, otherwise do not reject H 0 Not rejecting H 0 is NOT the same as accepting H 0 There are two types of errors: rejecting a true null (Type I) and not rejecting a false null (Type II) Summary

Statistics: Unlocking the Power of Data Lock 5 To Do Project 1 proposal due TODAY at 5pmproposal Read Section 4.3 Do Homework 4 (due Thursday, 10/4)Homework 4