1. Rejecting Chance – Testing Hypotheses in Research Thought Questions 1. Want to test a claim about the proportion of a population who have a certain trait. Collect data and discover that if claim true, the sample proportion you observed is so large that it falls at 99 th percentile of possible sample proportions for your sample size. Would you believe claim and conclude that you just happened to get a weird sample, or would you reject the claim? What if result was at 70 th percentile? At 99.99 th percentile? 2. Suppose half (0.50) of a population would answer yes when asked if support the death penalty. Random sample of 400 results in 220, or 0.55, who answer yes. Rule for Sample Proportions => potential sample proportions are approximately bell-shaped, with standard deviation (of sample proportion) of 0.025. The standardized score for observed value of 0.55 is z-score = 2. How often you would expect to see a standardized score at least that large or larger? 3. In the courtroom, juries must make a decision about the guilt or innocence of a defendant. Suppose you are on the jury in a murder trial. It is obviously a mistake if the jury claims the suspect is guilty when in fact he or she is innocent. What is the other type of mistake the jury could make? Which is more serious?

2. Rejecting Chance – Testing Hypotheses in Research Hypotheses and P-values The claim being tested in a statistical test is called the null hypothesis. The test is designed to assess the strength of the evidence against the null hypothesis. Example 1: :The proportion of all coffee drinkers who prefer fresh to instant coffee. null hypothesis: true proportion = 0.5 The statement that we suspect is true instead of the null hypothesis is called the alternative hypothesis. In this example, the alternative hypothesis could be that the majority of the population favor fresh coffee, that is alternative hypothesis: true proportion > 0.5 A significance test looks for evidence against the null hypothesis and in favor of the alternative hypothesis. Say we find 28 subjects in a sample of 50 that favor fresh coffee. How likely is this if in fact only half the population feel this way? Say we find 36 subjects in a sample of 50 that favor fresh coffee. A significance test answers this question by giving a probability: the probability of getting an outcome at least as far as the actually observed outcome from what we would expect when the null hypothesis is true.

3. Rejecting Chance – Testing Hypotheses in Research Example 1 Continued : Tasting Coffee The hypotheses: We want to test the hypotheses Null hypothesis: true proportion = 0.5 Alternative hypothesis: true proportion > 0.5 Here the true proportion is the proportion of the population of all coffee drinkers who prefer fresh coffee to instant coffee. The sampling distribution: If the null hypothesis is true, so that true proportion = 0.5, the distribution of sample proportion follows a Normal distribution centered at the true proportion=0.5 with standard deviation (of the sample proportion): (null value)(1 – null value) sample size = square root of (0.5)(0.5)/50 = 0.0707 The data: 1. A sample of 50 people found that 28 preferred fresh coffee. The sample proportion is equal to 0.56. 2. A second sample of 50 people found that 36 preferred fresh coffee. The sample proportion is equal to 0.72.

4. Rejecting Chance – Testing Hypotheses in Research To find any Normal curve probability, we calculate the standardized score (z-score). Remember, to calculate the standardized score for any observation is standardized score = observed value – null value/standard deviation 1. The standardized score ( known as the test statistic) for the sample proportion = 0.56 is z-score = 0.56 – 0.5/0.0707 =.84 2. The standardized score ( known as the test statistic) for the sample proportion = 0.72 is z-score = 0.72 – 0.5/0.0707 = 3.1 1. Table 8.1 says that the standardized score of.84 is the 80th percentile of a Normal distribution. 2. Table 8.1 says that the standardized score of 3.1 is the 99.9th percentile of a Normal distribution. Example 1 Continued : Tasting Coffee

5. Rejecting Chance – Testing Hypotheses in Research Hypotheses and P-values 1. We want the probability that 28 or more of 50 subjects would favor fresh coffee, given that the null hypothesis: true proportion = 0.5. 2. We want the probability that 36 or more of 50 subjects would favor fresh coffee, given that the null hypothesis: true proportion = 0.5. 1. The probability is 0.2 2. The probability is very small = 0.001 P-value The probability, computed assuming that the null hypothesis is true, that the sample outcome will be as extreme or more extreme than the actually observed outcome is called the P-value of the test. The smaller the P-value the stronger the evidence against the null hypothesis provided by the data.

6. Rejecting Chance – Testing Hypotheses in Research Hypotheses and P-values Example 2: The French naturalist Count Buffon (1707 – 1788) tossed a coin 4040 times. He got 2048 heads. The sample proportion of heads is 2048/4040 = 0.507 That's a bit more than one-half. Is this evidence that Buffon's coin was not balanced? The hypotheses. The null hypothesis says that the coin is balanced. We did not suspect a bias in a specific direction before we saw the data, so the alternative hypothesis is just "the coin is not balanced." The two hypotheses are Null Hypothesis: true proportion of heads equal to 0.5 Alternative Hypothesis : true proportion of heads not equal to 0.5 The sampling distribution: If the null hypothesis is true, the distribution of sample proportion follows a Normal distribution centered at the true proportion=0.5 with standard deviation (of the sample proportion): (null value)(1 – null value) sample size = square root of (0.5)(0.5)/4040 = 0.00787

7. Rejecting Chance – Testing Hypotheses in Research The data: Tossed a coin 4040 times. He got 2048 heads. The sample proportion of heads is 2048/4040 = 0.507 To find any Normal curve probability, we calculate the standardized score (z-score). To calculate the standardized score for any observation is standardized score = observed value – null value/standard deviation The standardized score ( known as the test statistic) for the sample proportion = 0.56 is standardized score (or z-score) = 0.507 – 0.5/0.00787 =.88 Table 8.1 says that the standardized score of.88 is the 81th percentile of a Normal distribution. Example 2 Continued : Count Buffon

8. Rejecting Chance – Testing Hypotheses in Research Example 2 Continue d - Count Buffon Coin Toss 0.50.507 Sampling Distribution of sample proportion Buffon’s sample proportion of heads falls just under one standard deviation above 0.5. The picture already suggests that this is not an unlikely outcome that would give strong evidence against the claim that true proportion equal to 0.5 The P-value: How unlikely is an outcome as far from 0.5 as Buffon's sample proportion = 0.507? Because the alternative hypothesis allows the true proportion to lie on either side of 0.5, values of sample proportion far from 0.5 in either direction provide evidence against the null hypothessis and in favor of the alternative. The P-value is therefore the probability that a sample proportion lies as far from 0.5 in either direction as the observed sample proportion equal to 0.507. The P-value is equal to 0.37

9. Rejecting Chance – Testing Hypotheses in Research The conclusion: A truly balanced coin would give a result this far or farther from 0.5 in 37% of all repetitions of Buffon's trial. His result gives no reason to think that his coin was not balanced. The alternative: true proportion greater than 0.5 in Example 1- tasting coffee is a one-sided alternative because the effect we seek evidence for says that the population proportion is greater than one-half. The alternative: true proportion not equal to 0.5 in Example 2 - Count Buffon Coin Toss is a two sided alternative because we ask only whether or not the coin is balanced. Example 2 Continue d - Count Buffon Coin Toss

10. Rejecting Chance – Testing Hypotheses in Research Statistical Significance We can decide in advance how much evidence against the alternative we will insist on. The way to do this is to say how small a P-value we require. When designing our study, if we choose a level of significance equal to 0.05, we are requiring that the data give evidence against the null hypothesis so strong that it would happen no more than 5% of the time (1 time in 20) when the null hypothesis is true. If we choose a level of significance equal to 0.01, we are insisting on stronger evidence against the null hypothesis, evidence so strong that it would appear only 1% of the time (1 time in 100) if the null hypothesis is in fact true. The significance level we choose is called the alpha level of significance. Statistical Significance: If the P-value is as small or smaller than the alpha level of significance, we say that the data are statistically significant at that particular level of significance. How to interpret high p-values (greater than 0.05) Imagine you shot baskets with Michael Jordan. He shot 7 straight free throws, you hit 3 and missed 4. You calculated a p-value of 0.07 for the null hypothesis that you shoot baskets as well he does.

11. Rejecting Chance – Testing Hypotheses in Research Summary of Basic Steps for Testing Hypotheses 1. Determine the null hypothesis and the alternative hypothesis. 2. Collect data and summarize with a single number called a test statistic. 3. Determine how unlikely test statistic would be if null hypothesis were true. p-value = probability of observing a standardized score as extreme or more extreme (in the direction specified in the alternative hypothesis) if the null hypothesis is true. 4. Make a decision. Researcher compares the p-value to what is known as the alpha level of significance. If the p-value is greater than the level of significance: Fail to reject the null hypothesis If the p-value is less than or equal to the level of significance: Reject the null hypothesis in favor of the alternative hypothesis

12. Example 3: Family Structure in Teen Survey Rejecting Chance – Testing Hypotheses in Research Government reports 67% of teens live with both parents but survey gave 84% => does survey population differ? Null hypothesis: For the population of teens represented by the survey, the proportion living with both parents is 0.67. Alternative hypothesis: For population of teens represented by survey, proportion living with both parents is not equal to 0.67. Survey of 1,987 teens => 84% living with both parents. So the standard deviation = (0.67)  (1 – 0.67) = 0.0105. 1987 Test statistic: z = observed value – null value = 0.84 – 0.67 = 16 (extremely large!) standard deviation 0.0105 Recall alternative hypothesis was two-sided. So p-value = 2  [proportion of bell-shaped curve above 16]. Table 8.1 => proportion is essentially 0. Almost impossible to observe a sample of 1,987 teens with 84% living with both parents if only 67% of population do.

13. Rejecting Chance – Testing Hypotheses in Research Example: Suppose you think that the proportion of coupon users in grocery stores in your town has Decreased from 10 years ago. You know from previous research that 10 years ago, 35% of grocery store customers in your town used coupons. Suppose you take a random sample of 100 customers from a variety of grocery stores in your town, and you find that 25 percent of them use coupons. What are the null and alternative hypotheses? z-score = 0.25– 0.35/0.046 = -2.17 The p-value for the observed results is.0179. What is your conclusion?

14. What Can Go Wrong: The Two Types of Errors Type 1 error: Rejecting the null hypothesis when it is actually true Type 2 error: Failing to reject the null hypothesis when it is actually false Courtroom Analogy: Potential choices and errors Null hypothesis: Defendant is innocent. Alternative hypothesis: Defendant is guilty. Choice A: We cannot rule out that defendant is innocent, so he or she is set free without penalty. Potential error: A criminal has been erroneously freed. Choice B: We believe enough evidence to conclude the defendant is guilty. Potential error: An innocent person falsely convicted and guilty party remains free. Rejecting Chance – Testing Hypotheses in Research

15. As another example, imagine a statistical test in terms of a diagnostic test for a disease. The table shows that when you run a statistical test, you can only make one of two errors: You can reject the null hypothesis when it is true (a false positive – Type I error) OR You fail to reject the null hypothesis when it is false (a false negative – Type II error). Another name is (alpha) error for false positive and (beta) error for false negative. Type I (or alpha) error is the level of significance we choose for our statistical test. Null hypothesis: You do not have the disease Alternative: You have the disease

16. Rejecting Chance – Testing Hypotheses in Research Probabilities Associated with Errors We can only specify the conditional probability of making a type I error, given that the null hypothesis is true. That probability is called the alpha level of significance. A type II error is made if the alternative hypothesis is true, but you fail to choose it. The probability of doing that depends on which part of the alternative hypothesis is true, so computing the probability of making a type II error is not feasible. Example In coffee tasting example, if only 51% of all coffee drinkers prefer fresh coffee, then a sample of 50 coffee tasters could result in a sample proportion that would lead to not rejecting the null hypothesis when it is in fact false (easily make a type II error). However, if 90% of all coffee drinkers prefer fresh coffee, the sample proportion will almost surely be large enough to reject the null hypothesis in favor of the alternative(very low probability of making a type II error). Problem: Impossible to know what part of the alternative (ex. 51% or 90%) is true.

17. Rejecting Chance – Testing Hypotheses in Research The Power of a Test The power of a test is the probability of making the correct decision when the alternative hypothesis is true. Medical Testing Example Detecting that a person has the disease when they truly have it, a true positive Coffee Tasting Example The ability to detect that a majority prefer fresh coffee will be much higher if that majority constitutes 90% than if it constitutes just 51% of the population. If the population value falls close to the value specified in null hypothesis, then it is difficult to get enough evidence from the sample to conclusively choose the alternative hypothesis. There will be a relatively high probability of making a type 2 error, and the test will have relatively low power in that case.

18. Type I, Type II errors and Power of a Test Rejecting Chance – Testing Hypotheses in Research

19. Example : Experiment in ESP Experiment: Subject tried to guess which of four videos the “sender” was watching in another room. Null hypothesis: Results due to chance guessing so probability of success is 0.25. Alternative hypothesis: Results not due to chance guessing, so probability of success is higher than 0.25. Sample proportion of successes: 122/355 = 0.344. The standard deviation = (0.25)  (1 – 0.25) = 0.023. 355 Test statistic: z-score = (0.344 – 0.25)/0.023 = 4.09 The p-value is about 0.00005. What conclusion do we make?

20. Rejecting Chance – Testing Hypotheses in Research Text Questions