Lecture 10 PY 427 Statistics 1 Fall 2006 Kin Ching Kong, Ph.D Chicago School of Professional Psychology Lecture 10 Kin Ching Kong, Ph.D

Agenda Analysis of Variance (continue) The Distribution of F-Ratios Review Intro. to ANOVA Hypotheses for ANOVA The Test Statistic for ANOVA: F The Logic of Analysis of Variance ANOVA Notation & Formulas The Distribution of F-Ratios Hypothesis Testing Measuring Effect Size Assumptions for the independent-measure ANOVA Post Hoc Tests

Analysis of Variance (ANOVA) ANOVA is used to compare two or more means The Question: Does the mean differences observed among the samples reflect mean differences among the populations? Two Possibilities: There really are no differences in the population means, the observed differences are due to chance (sampling error). The population means are truly different, and are partly responsible for differences in the sample means. Figure 13.1

Hypotheses for ANOVA The experiment: learning performance under three temperature: 50o, 70o, 90o Design: single-factor, between-subject (or independent-measures. The Hypotheses: H0: m1 = m2 = m3 (no differences in pop. means) H1: At least one pop. mean is different from the others, (or not all the pop. means are equal). There are many possible specific alternative hypotheses (e.g. all 3 means are different, first two means are identical but the third is different, the first is different and last two identical, etc.) So far we talked about inferential statistics using one sample to make inferences about an unknown population.

The Test Statistic for ANOVA The Test Statistic for t tests: t = obtained difference between sample means difference expected by chance (error) The Test Statistic for ANOVA, F-ratio: F = variance (differences) between sample means variance (differences) expected by chance (error) The F-ratio is based on variance rather than means Problem: how to define & calculate mean differences when there are more than two means. Solution: use variance to define and measure the size of differences among the sample means. E.g M1 = 20, M2 = 30, M3 =35 s2 = 58.33 M1 = 28, M2 = 30, M3 = 31 s2 = 2.33

The Logic of Analysis of Variance, The Data The experiment: learning under 3 different temperature The design: single factor between-subjects The data: DV = # of problems solved correctly. Temp. 50O Temp. 70O 4 1 3 2 6 M = 1 M = 4

The Logic of Analysis of Variance The Goal of ANOVA Measure the amount of variability in a data set and explain where it comes from. Step I: Measure Total Variance The variability in the whole data set (scores from all samples combined) Step II: Partition Total Variance into two components: Between-Treatment Variance Differences between treatment conditions Within-Treatment Variance: Differences within each treatment condition

The Between-Treatments Variance Measures how much difference exists between the treatment conditions (i.e. differences among treatment means). Two Sources For The Differences Between Treatments: Treatment effects Chance (unplanned & unpredictable difference) Individual differences Experimental error (Measurement error) To Demonstrate There Really Is A Treatment Effect: Show that differences between treatments are bigger than would be expected by chance alone.

The Within-Treatment Variance Measures differences due to chance, or when there is no treatment effect, H0 is true. Figure 13.2 Analysis of Variance

The F-Ratio: Test Statistic for ANOVA The F-Ratio compares the two component variance: F = Variance between treatments Variance within treatments = actual differences between treatments differences expected with no treatment effect F = treatment effect + differences due to chance differences due to chance (error) When there is no treatment effect: F = 0 + differences due to chance F is close to 1 differences due to chance When there is a treatment effect: The numerator significantly > the denominator F significantly > 1

ANOVA Notations SX2 = 106 G = 30 N = 15 k = 3 T1 = 5 T2 = 20 T3 = 5 SS1 = 6 SS2 = 6 SS3 = 4 n1 = 5 n2 = 5 n3 = 5 M1 = 1 M2 = 4 M3 = 1 Temp. 50O Temp. 70O 4 1 3 2 6 M = 1 M = 4

ANOVA Notations (Continue) k = number of treatment conditions (or number of levels of a factor) ni = number of scores in treatment i (i = 1 to k) N = total number of scores in the entire study. Ti = The total (SX) for treatment condition I G = The Grand Total = SX for all the scores, or G = ST For a one sample t test, the statistic is M, so the standard error is sM

ANOVA Formulas ANOVA Summary Table: Source SS df MS Between treatments F = MSbetween Within treatments MSwithin Total F = Variance between treatments Variance within treatments Variance, s2 = SS/df =MS F = MSbetween MSwithin To fill in the ANOVA summary table, need to calculate nine values: 3 SS, 3 df, 2 MS and F

ANOVA Formulas: SS Total Sum of Squares: SStotal = SX2 – (SX)2 = SX2 – G2 N N For our example: SStotal = SX2 – G2 = 106 -302/15 = 106 – 60 = 46 N Within-Treatment Sum of Squares: SSwithin = SSSwithin each treatment For our example: SSwithin = 6 + 6 + 4 = 16 Between-Treatment Sum of Squares: SSbetween = S T2 - G2 n N SSbetween = S T2 - G2 = 52 + 202 + 52 – 302 = 5 + 80 +5 -60 =30 n N 5 5 5 15

ANOVA Formulas, df Total Degrees of Freedom: dftotal = N - 1 For our example, dftotal = 15 – 1 = 14 Within-Treatment Degrees of Freedom: dfwithin = Sdfin each treatment = S(n-1) = N - k For our example, dfwithin = 15 – 3 = 12 Between-Treatment Degrees of Freedom: dfbetween = k – 1 For our example, dfbetween = 3 – 1 = 2

ANOVA Formulas, MS & F-Ratio Variance Between-Treatment, MSbetween: s2 = MSbetween = SSbetween/dfbetween For our example, MSbetween = 30/2 = 15 Variance Within-Treatment, MSwithin: s2 = MSwithin = SSwithin/dfwithin For our example, MSwithin = 16/12 = 1.33 The F-Ratio: F = MSbetween MSwithin For our example: F = 15/1.33 = 11.28

ANOVA Formulas Source SS df MS ANOVA Summary Table: Between treatments 30 2 15 F = 11.28 Within treatments 16 12 1.33 Total 46 14

The Distribution of F-Ratios Two characteristics of F values: F values are always positive because variances are always positive. When H0 is true, the numerator and denominator of the F-ratio estimate the same variance, thus, the ratio should be near 1. In other words, the distribution of F-ratios should pile up around 1.00 The Distribution of F-ratios: Cut off at zero (all positive values) Piles up around 1.00 Tapers off to the right. The exact shape of the F distribution depends on the df’s in the two variances. Figure 13.6 of your book

The F Distribution Table Table B.4 Find df of the numerator in first row. Find df of the denominator in first column The intersection of these two df’s is a pair of numbers: The smaller number is the critical value for a = .05 The larger number is the critical value for a = .01 Table 13.3 E.g. F = 4.18 with df = 2, 15. Is this value sufficient to reject H0 with a = .05? a =.01?

Hypothesis Testing (the experiment) Research Goal: Evaluate the effectiveness of three pain relievers (A, B, C) and a placebo. Experiment: Participants: four groups, n = 5 in each group Treatment (IV): Drug A, B, C and placebo Design: Single-factor, repeated-measures Dependent Variable (DV): Amount of time participants can withstand a painfully hot stimulus.

Hypothesis Testing (Data) SX2 = 262 T = 5 T = 10 T = 20 T = 25 SS = 8 SS = 8 SS = 6 SS = 10 Placebo Drug A Drug B Drug C 3 4 6 7 2 1 5

ANOVA Summary Table ANOVA Summary Table: Source SS df MS Between treatments F = Within treatments Total

ANOVA Calculations dftotal = N – 1 = 20 – 1 = 19 dfbetween = k – 1 = 4 – 1 =3 dfwithin = N – k = 16 SStotal = SX2 – G2/N = 262 – 602/20 = 82 SSwithin = SSSinside each treatment= 8 + 8 + 6 + 10 = 32 SSbetween = S T2/n – G2/N = 52/5 + 102/5 + 202/5 +252/5 – 602/20 = 50 MSbetween = SSbetween/ dfbetween = 50/3 = 16.67 MSwithin = SSwithin/ dfwithin = 32/16 = 2.00 F = MSbetween/ MSwithin = 16.67/2.00 = 8.33

ANOVA Formulas Source SS df MS ANOVA Summary Table: Between treatments 50 3 16.67 F = 8.33 Within treatments 32 16 2.00 Total 82 19

Hypothesis Testing with ANOVA Step 1: State the Hypotheses: H0: m1 = m2 = m3 = m4 (there is no treatment effect) H1: At least one of the treatment means are different. The level of significant is set at a = .05 Step 2: Locate the Critical Region: df = 3, 16, Fcritical = 3.24 Figure 13.7 Step 3: Calculate the test statistic: F = MSbetween / MSwithin = 8.33 Step 4: Make a decision: Since the test statistic, F = 8.33 falls in the critical region, reject H0 and concludes that there is a significant treatment effect.

Measuring Effect Size for ANOVA A significant difference Means that the difference observed in the samples is very unlikely to have occurred just by chance. Statistical significant does not necessarily mean large effect. Measuring effect size for ANOVA: r2: the percentage of variance accounted for by treatment r2 = SSbetween SStotal In published reports, the r2 value for ANOVA is usually call h2 (the Greek letter eta squared) For our example, h2 = 50/82 = 0.61 The pooled variance is used instead of the individual sample variances.

Assumptions Assumptions for the Independent-Measure ANOVA: The observations within each sample must be independent. The populations from which the samples are selected must be normal. The populations from which the samples are selected must have equal variances (homogeneity of variance)

Post Hoc Tests, Intro A significant F-ratio: Indicate that a significant difference exit, that not all the means are equal. Does not indicate which means are different and which are not. Example: M1 = 3, M2 = 5, M3 = 10 M2 – M1 = 2, M3 – M2 = 5, M3 – M1 = 7 A significant F indicate that at least one of these differences are significant, M3 = M1, but what about the other two? Post hoc tests are used to find out which of these difference are significant.

Post Hoc Tests & Type I Error are additional hypothesis test that are done after an analysis of variance revealed a significant difference They are performed to determine exactly which mean differences are significant and which are not. Post hoc test and Type I Error Post hoc tests compare two means at a time, i.e. pairwise comparisons. The process involve performing a series of separate hypothesis tests. Each of these tests includes the risk of a Type I error With more tests, the risk of a type I error accumulates Experimentwise alpha level: the overall probability of a Type I Error that accumulates over a series of separate hypothesis tests.

Post Hoc Tests, Planned Comparisons Controlling Type I Error (experimentwise alpha level) Whenever more than one test is done, need to be concerned about the experimentwise Type I Error. Planned Comparisons Planned Comparisons: specific mean differences are predicted by specific hypotheses before the study is conducted. Because a few specific comparisons were planned before the data were collected, many statisticians argue that planned comparisons can be conducted with a standard alpha, without concern about inflating the risk of a Type I error. Dunn Test: It is often recommended that researchers protect against an inflated alpha level by dividing alpha equally among the planned comparisons.

Post Hoc Tests, Tukey’s HSD Unplanned Comparisons sifting through the data by conducting a large number of comparisons. Tukey’s Honestly Significant Difference (HSD) Compute a single value (HSD) that determines the minimum difference that is necessary for significance. If a pairwise difference is > Tukey’s HSD, you conclude that there is a significant difference between the two means. HSD = q n = number of scores in each treatment, Tukey’s HSD test requires equal n’s Table B.5 to find value of q. (k = number of treatment conditions, df error term = df for the denominator of the F-ratio)

Post Hoc Tests, Tukey’s HSD, Example M1 = 3.00 M2 = 5.44 M3 = 7.00 ANOVA Summary Table Source SS df MS Between 73.19 2 36.60 F (2, 24) = 9.15 Within 96.00 24 4.00 Total 169.19 26 HSD = q = 3.53 = 2.36 M2 – M1 = 2.44, significant M3 – M1 = 4.00, significant M3 – M2 = 1.56, nonsignificant