University of Durham D Dr Robert Coe University of Durham School of Education Tel: (+44 / 0) Fax: (+44 / 0) Inferences from Data Doctor of Education (EdD) Analysing, Interpreting and Using Educational Research (Research Methodology)

© 2005 Robert Coe, University of Durham 2 Statistical significance Could the difference between the two groups have arisen purely as a result of sampling variation? Yes, always (but may be unlikely) NOT: ‘Is the result real?’ or ‘Would it be replicated?’ Statistical significance (p-value) tells you the probability such a difference would have been found by pure chance Often take p = 0.05 as critical But this is arbitrary

© 2005 Robert Coe, University of Durham 3 A Normal population Population mean = 100, S.D. = 15

© 2005 Robert Coe, University of Durham 4 Random samples from a Normal population Difference between the two means = 97.0 – 98.3 = -1.3

© 2005 Robert Coe, University of Durham pairs of samples … 63 out of the 100 differences are bigger than 1.31 (So this is not at all extreme)

© 2005 Robert Coe, University of Durham 6 T-test Test for statistical significance of difference between two means If two samples of the same size were drawn independently and at random from Normal populations with equal standard deviations, how likely is it that their means would differ by as much as these two? Calculate ‘t’ (similar to z-score for Normal dist) ‘t’ is proportional to ES x  n (n = no. in each group)

© 2005 Robert Coe, University of Durham 7 Conditions for use of t-test Independence Use paired samples test if paired (eg test, retest for same subjects) Cluster sampling violates this Random samples This condition is seldom met From Normal distribution Unlikely to be a problem unless heavily skewed Robust if n > 30 Equal std deviations (homoscedasticity) Robust if n 1  n 2 Can test for equality (alternative t-test if unequal) Interval scale Non-linear transformations will affect t-test Mann-Whitney U-test can be used instead of the t-test without making these 3 assumptions

© 2005 Robert Coe, University of Durham 8 Other significance tests Chi-squared (  2 ) test Tests the difference between observed and expected frequencies Often used to test the independence of two category variables (e.g. male/female and pass/fail) Analysis of Variance (ANOVA) Compares means of more than two groups Lots of variants: ANCOVA, 2-way ANOVA, MANOVA All are equivalent to (multiple) regression

© 2005 Robert Coe, University of Durham 9 Accumulation of significance tests Six replications of an experiment comparing children’s learning in the morning and afternoon: One shows a significant difference (afternoon is better), five show no significant difference Overall conclusion: no clear difference

© 2005 Robert Coe, University of Durham 10 Accumulation of effect size estimates Overall conclusion: significant benefit for afternoon

© 2005 Robert Coe, University of Durham 11 Hypersensitive significance tests Statistically significant difference THE TREATMENT WORKED! Difference not significant IT DIDN’T WORK!

© 2005 Robert Coe, University of Durham 12 Effect Size vs Statistical Significance Emphasises amounts, not just directions Avoids inappropriate dichotomies Avoids confusion over ‘significance’ Draws attention to power Avoids ‘file drawer’ problem Promotes synthesis rather than disagreement Allows accumulation of knowledge

© 2005 Robert Coe, University of Durham 13 Advice on significance testing Always look at the data Don’t say ‘significant’ when you mean ‘statistically significant’ Always give p-value, not just ‘sig’ or ‘non-sig’ Better still, use confidence interval around estimate – contains same info Beware of multiple and post-hoc comparisons Replication is the only guarantee of replication Don’t ‘count votes’ (no. of studies sig/not) Combine results using meta-analysis