Significance Testing Chapter 13 Victor Katch Kinesiology

Critical Region The critical region (or rejection region) is the set of all values of the test statistic that cause us to reject the null hypothesis. For example, see the red-shaded region in previous Figure. Victor Katch Kinesiology

Significance Level The significance level (denoted by ) is the probability that the test statistic will fall in the critical region when the null hypothesis is actually true. Common choices for  are 0.05, 0.01, and 0.10. Victor Katch Kinesiology

Critical Value A critical value is any value separating the critical region (where we reject the H0) from the values of the test statistic that does not lead to rejection of the null hypothesis, the sampling distribution that applies, and the significance level . For example, the critical value of z = 1.645 corresponds to a significance level of  = 0.05. Victor Katch Kinesiology

Two-tailed, Right-tailed, Left-tailed Tests The tails in a distribution are the extreme regions bounded by critical values. page 373 of text Victor Katch Kinesiology

 is divided equally between the two tails of the critical Two-tailed Test H0: = H1:   is divided equally between the two tails of the critical region Means less than or greater than Victor Katch Kinesiology

Right-tailed Test H0: = H1: > Points Right Victor Katch Kinesiology

Left-tailed Test H0: = H1: < Points Left Victor Katch Kinesiology

P-Value The P-value (or p-value or probability value) is the probability of getting a value of the test statistic that is at least as extreme as the one representing the sample data, assuming that the null hypothesis is true. The null hypothesis is rejected if the P-value is very small, such as 0.05 or less. Victor Katch Kinesiology

Conclusions in Hypothesis Testing We always test the null hypothesis. 1. Reject the H0 2. Fail to reject the H0 page 374 of text. Examples at bottom of page and top of page 375 Victor Katch Kinesiology

Accept versus Fail to Reject Some texts use “accept the null hypothesis.” We are not proving the null hypothesis. The sample evidence is not strong enough to warrant rejection (such as not enough evidence to convict a suspect). page 374 of text The term ‘accept’ is somewhat misleading, implying incorrectly that the null has been proven. The phrase ‘fail to reject’ represents the result more correctly. Victor Katch Kinesiology

Decision Criterion Traditional method: Reject H0 if the test statistic falls within the critical region. Fail to reject H0 if the test statistic does not fall within the critical region. Victor Katch Kinesiology

Decision Criterion P-value method: Reject H0 if P-value   (where  is the significance level, such as 0.05). Fail to reject H0 if P-value > . Victor Katch Kinesiology

Decision Criterion Another option: Instead of using a significance level such as 0.05, simply identify the P-value and leave the decision to the reader. Victor Katch Kinesiology

Example: Finding P-values Victor Katch Kinesiology

Wording of Final Conclusion Victor Katch Kinesiology

Hypothesis testing about: a population mean or mean difference (paired data) the difference between means of two populations the difference between two population proportions Three Cautions: 1. Inference is only valid if the sample is representative of the population for the question of interest. 2. Hypotheses and conclusions apply to the larger population(s) represented by the sample(s). 3. If the distribution of a quantitative variable is highly skewed, consider analyzing the median rather than the mean – called nonparametric methods. Victor Katch Kinesiology

Significance Testing Steps in Any Hypothesis Test Determine the null and alternative hypotheses. Verify necessary data conditions, and if met, summarize the data into an appropriate test statistic. Assuming the null hypothesis is true, find the p-value. Decide whether or not the result is statistically significant based on the p-value. Report the conclusion in the context of the situation. Victor Katch Kinesiology

Testing Hypotheses About One Mean or Paired Data Step 1: Determine null and alternative hypotheses 1. H0: m = m0 versus Ha: m  m0 (two-sided) 2. H0: m  m0 versus Ha: m < m0 (one-sided) 3. H0: m  m0 versus Ha: m > m0 (one-sided) Often H0 for a one-sided test is written as H0: m = m0. Remember a p-value is computed assuming H0 is true, and m0 is the value used for that computation. Victor Katch Kinesiology

Step 2: Verify Necessary Data Condition Situation 1: Population of measurements of interest is approximately normal, and a random sample of any size is measured. In practice, use method if shape is not notably skewed or no extreme outliers. Situation 2: Population of measurements of interest is not approximately normal, but a large random sample (n  30) is measured. If extreme outliers or extreme skewness, better to have a larger sample. Victor Katch Kinesiology

Continuing Step 2: The Test Statistic The t-statistic is a standardized score for measuring the difference between the sample mean and the null hypothesis value of the population mean: This t-statistic has (approx) a t-distribution with df = n - 1. Victor Katch Kinesiology

Step 3: Assuming H0 true, Find the p-value For H1 less than, the p-value is the area below t, even if t is positive. For H1 greater than, the p-value is the area above t, even if t is negative. For H1 two-sided, p-value is 2  area above |t|. Victor Katch Kinesiology

Steps 4 and 5: Decide Whether or Not the Result is Statistically Significant based on the p-value and Report the Conclusion in the Context of the Situation These two steps remain the same for all of the hypothesis tests. Choose a level of significance a, and reject H0 if the p-value is less than (or equal to) a. Otherwise, conclude that there is not enough evidence to support the alternative hypothesis. Victor Katch Kinesiology

Example Normal Body Temperature What is normal body temperature? Is it actually less than 98.6 degrees Fahrenheit (on average)? Step 1: State the null and alternative hypotheses H0: m = 98.6 Ha: m < 98.6 where m = mean body temperature in human population. Victor Katch Kinesiology

Example Normal Body Temp (cont) Data: random sample of n = 18 normal body temps 98.2 97.8 99.0 98.6 98.2 97.8 98.4 99.7 98.2 97.4 97.6 98.4 98.0 99.2 98.6 97.1 97.2 98.5 Step 2: Verify data conditions … x no outliers nor strong skewness. Sample mean of 98.217 is close to sample median of 98.2. Victor Katch Kinesiology

Example Normal Body Temp (cont) Step 2: … Summarizing data with a test statistic Test of mu = 98.600 vs mu < 98.600 Variable N Mean StDev SE Mean T P Temperature 18 98.217 0.684 0.161 -2.38 0.015 Key elements: Sample statistic: = 98.217 (under “Mean”) Standard error: (under “SE Mean”) (under “T”) Victor Katch Kinesiology

Example Normal Body Temp (cont) Step 3: Find the p-value From output: p-value = 0.015 From Table A.3: p-value is between 0.016 and 0.010. Area to left of t = -2.38 equals area to right of t = +2.38. The value t = 2.38 is between column headings 2.33 and 2.58 in table, and for df =17, the one-sided p-values are 0.016 and 0.010. Victor Katch Kinesiology

Example Normal Body Temp (cont) Step 4: Decide whether or not the result is statistically significant based on the p-value Using a = 0.05 as the level of significance criterion, the results are statistically significant because 0.015, the p-value of the test, is less than 0.05. In other words, we can reject the null hypothesis. Step 5: Report the Conclusion We can conclude, based on these data, that the mean temperature in the human population is actually less than 98.6 degrees. Victor Katch Kinesiology

Paired Data and the Paired t-Test Data: two variables for n individuals or pairs; use the difference d = x1 – x2. Parameter: md = population mean of differences Sample estimate: = sample mean of the differences Standard deviation and standard error: sd = standard deviation of the sample of differences; Often of interest: Is the mean difference in the population different from 0? Victor Katch Kinesiology

Steps for a Paired t-Test Step 1: Determine null and alternative hypotheses H0: md = 0 versus Ha: md  0 or Ha: md < 0 or Ha: md > 0 Watch how differences are defined for selecting the Ha. Step 2: Verify data conditions and compute test statistic Conditions apply to the differences. The t-test statistic is: Steps 3, 4 and 5: Similar to t-test for a single mean. The df = n – 1, where n is the number of differences. Victor Katch Kinesiology

Example Effect of Alcohol Study: n = 10 pilots perform simulation first under sober conditions and then after drinking alcohol. Response: Amount of useful performance time. (longer time is better) Question: Does useful performance time decrease with alcohol use? Step 1: State the null and alternative hypotheses H0: md = 0 versus Ha: md > 0 where md = population mean difference between alcohol and no alcohol measurements if all pilots took these tests. Victor Katch Kinesiology

Example Effect of Alcohol (cont) Data: random sample of n = 10 time differences Step 2: Verify data conditions … Boxplot shows no outliers nor extreme skewness. Victor Katch Kinesiology

Example Effect of Alcohol (cont) Step 2: … Summarizing data with a test statistic Test of mu = 0.0 vs mu > 0.0 Variable N Mean StDev SE Mean T P Diff 10 195.6 230.5 72.9 2.68 0.013 Key elements: Sample statistic: = 195.6 (under “Mean”) Standard error: (under “SE Mean”) (under “T”) Victor Katch Kinesiology

Example Effect of Alcohol (cont) Step 3: Find the p-value From output: p-value = 0.013 From Table A.3: p-value is between 0.007 and 0.015. The value t = 2.68 is between column headings 2.58 and 3.00 in the table, and for df =9, the one-sided p-values are 0.015 and 0.007. Victor Katch Kinesiology

Example Effect of Alcohol (cont) Steps 4 and 5: Decide whether or not the result is statistically significant based on the p-value and Report the Conclusion Using a = 0.05 as the level of significance criterion, we can reject the null hypothesis since the p-value of 0.013 is less than 0.05. Even with a small experiment, it appears that alcohol has a statistically significant effect and decreases performance time. Victor Katch Kinesiology

Testing The Difference between Two Means (Independent Samples) Step 1: Determine null and alternative hypotheses H0: m1 – m2 = 0 versus Ha: m1 – m2  0 or Ha: m1 – m2 < 0 or Ha: m1 – m2 > 0 Watch how Population 1 and 2 are defined. Step 2: Verify data conditions and compute test statistic Both n’s are large or no extreme outliers or skewness in either sample. Samples are independent. The t-test statistic is: Steps 3, 4 and 5: Similar to t-test for one mean. Victor Katch Kinesiology

Example Effect of Stare on Driving Randomized experiment: Researchers either stared or did not stare at drivers stopped at a campus stop sign; Timed how long (sec) it took driver to proceed from sign to a mark on other side of the intersection. Question: Does stare speed up crossing times? Step 1: State the null and alternative hypotheses H0: m1 – m2 = 0 versus Ha: m1 – m2 > 0 where 1 = no-stare population and 2 = stare population. Victor Katch Kinesiology

Example Effect of Stare (cont) Data: n1 = 14 no stare and n2 = 13 stare responses Step 2: Verify data conditions … No outliers nor extreme skewness for either group. Victor Katch Kinesiology

Example Effect of Stare (cont) Step 2: … Summarizing data with a test statistic Sample statistic: = 6.63 – 5.59 = 1.04 seconds Standard error: Victor Katch Kinesiology

Example Effect of Stare (cont) Steps 3, 4 and 5: Determine the p-value and make a conclusion in context. The p-value = 0.013, so we reject the null hypothesis, the results are “statistically significant”. The p-value is determined using a t-distribution with df = 21 (df using Welch approximation formula) and finding area to right of t = 2.41. Table A.3 => p-value is between 0.009 and 0.015. We can conclude that if all drivers were stared at, the mean crossing times at an intersection would be faster than under normal conditions. Victor Katch Kinesiology

The Two Types of Errors and Their Probabilities When the null hypothesis is true, the probability of a type 1 error, the level of significance, and the a-level are all equivalent. When the null hypothesis is not true, a type 1 error cannot be made. Victor Katch Kinesiology

Type I Error A Type I error is the mistake of rejecting the null hypothesis when it is true. The symbol (alpha) is used to represent the probability of a type I error. Example on page 375 of text Victor Katch Kinesiology

Type II Error A Type II error is the mistake of failing to reject the null hypothesis when it is false. The symbol (beta) is used to represent the probability of a type II error. Victor Katch Kinesiology

Example: Assume that we a conducting a hypothesis test of the claim p > 0.5. Here are the null and alternative hypotheses: H0: p = 0.5, and H1: p > 0.5. a) Identify a type I error. b) Identify a type II error. Victor Katch Kinesiology

Example: Assume that we a conducting a hypothesis test of the claim p > 0.5. Here are the null and alternative hypotheses: H0: p = 0.5, and H1: p > 0.5. Identify a type I error. A type I error is the mistake of rejecting a true null hypothesis, so this is a type I error: Conclude that there is sufficient evidence to support p > 0.5, when in reality p = 0.5. Victor Katch Kinesiology

Example: Assume that we a conducting a hypothesis test of the claim p > 0.5. Here are the null and alternative hypotheses: H0: p = 0.5, and H1: p > 0.5. Identify a type II error A type II error is the mistake of failing to reject the null hypothesis when it is false, so this is a type II error: Fail to reject p = 0.5 (and therefore fail to support p > 0.5) when in reality p > 0.5. Victor Katch Kinesiology

Type I and Type II Errors Victor Katch Kinesiology

Controlling Type I and Type II Errors For any fixed , an increase in the sample size n will cause a decrease in  For any fixed sample size n , a decrease in  will cause an increase in . Conversely, an increase in  will cause a decrease in  . To decrease both  and , increase the sample size. page 377 in text Victor Katch Kinesiology

Power of a Hypothesis Test Definition Power of a Hypothesis Test The power of a hypothesis test is the probability (1 - ) of rejecting a false null hypothesis, which is computed by using a particular significance level  and a particular value of the population parameter that is an alternative to the value assumed true in the null hypothesis. Victor Katch Kinesiology

Trade-Off in Probability for Two Errors There is an inverse relationship between the probabilities of the two types of errors. Increase probability of a type 1 error => decrease in probability of a type 2 error Victor Katch Kinesiology

Type 2 Errors and Power Three factors that affect probability of a type 2 error 1. Sample size; larger n reduces the probability of a type 2 error without affecting the probability of a type 1 error. 2. Level of significance; larger a reduces probability of a type 2 error by increasing the probability of a type 1 error. 3. Actual value of the population parameter; (not in researcher’s control. Farther truth falls from null value (in Ha direction), the lower the probability of a type 2 error. When the alternative hypothesis is true, the probability of making the correct decision is called the power of a test. Victor Katch Kinesiology