Statistical Power: Sample & Effect Size Dr James Betts Developing Study Skills and Research Methods (HL20107)

Lecture Outline: Statistical Power Effect Size Smallest Worthwhile Effect Coefficient of Variation Sample Size Estimation.

Statistical Power Our consideration of post-hoc tests last week focused heavily on controlling type I error rate However, it is also of importance that the rate of type II errors is controlled Statistical power reflects the sensitivity of our test (i.e. the power to detect a genuine effect 80% of the time) So what dictates the power of a test?

Statistical Power Significance tests determine a P value based upon 3 factors: – – – The power of a test is therefore dictated by the balance between the number of subjects recruited and the Effect Size (ES).

ES AKA Cohens d (d=D/SD) SubjectPre-ScorePost-ScoreDifference Tom12164 Dick14173 Harry10122 James12153 Mean12153 SD …BUT see Dunlap et al. (1996) Psychological Methods 1 (2) p

ES Interpretation/Application Effect size shows us the magnitude of our effect relative to SD Based upon the magnitude of correlation between trials, Jacob Cohen suggests thresholds of >0.2 (small), >0.5 (moderate) & >0.8 (large) –(n.b. others favour >0.2, >0.6 & >1.2) So effect size provides a useful tool for examining differences irrespective of sample size Another major application of ES is therefore to determine the required sample size for our study.

GB = s USA = s Smallest Worthwhile Effect D = 0.01 s

Smallest Worthwhile Effect It would appear that even a small amount of primary variance from an ergogenic aid would guarantee victory to either competitor… …however, the error variance is such that a re-run could produce entirely different results…

…for an effect to guarantee first place, it would need to exceed the opponents time by more than his error variance. Re-Run

Coefficient of Variation (CV) The coefficient of variation expresses within subject variation as a % of their average performance: –e.g. USA test-retest form last 10 training sessions s s s s s s s s s s Mean = SD = CV =

Smallest Worthwhile Effect So when conducting applied research into performance enhancement, the smallest worthwhile effect can be based on the actual % improvement that produces a worthwhile increase in your chance of winning the event –example for 100 m sprint an improvement of 0.3 of CV converts 2 nd1 st once every ten races Hopkins et al. (1999) MSSE 31 (3) p However, we dont always have such ecologically valid data to support our laboratory investigations.

Smallest Worthwhile Effect Ideally, it is recommended that a pilot study is conducted so that the typical effect size can be established and A Priori sample size calculations can be conducted Alternatively, the rationale for a planned study is often supported by previously published literature, in which case this data can be used as a guide to the magnitude of effects which can be expected.

Sample Size Estimation Overall, published data can be used for A Priori power analysis as a general guide for how many subjects to recruit Then post-hoc power analysis can be conducted to calculate the actual statistical power given the sample size attained –e.g. Using similar supplements to those under investigation in the present study, van Loon et al. (2000) reported the inclusion of protein to accelerate muscle glycogen resynthesis by 18.8 mmol glucosyl units kg dry mass -1 h -1, with a pooled standard deviation of 6.6 mmol glucosyl units kg dry mass -1 h -1. Based upon these data it was estimated that a sample size of 6 has a 99% power to detect such differences.

The purpose of sample size formulae is not to give an exact number…but rather to subject the study design to scrutiny, including an assessment of the validity and reliability of data collection, and to give an estimate to distinguish whether tens, hundreds, or thousands of participants are required Williamson et al. (2000) JRSS 163: p. 10

Summary The power of a statistical test is influenced by the size of the effect and sample size Effect size provides a useful tool for examining data when sample size is small The smallest worthwhile effect can also be applied to determine how many subjects would be required for statistical significance Remember that our choice of data for this analysis was very subjective in places.