Introduction to Power and Effect Size  More to life than statistical significance  Reporting effect size  Assessing power

Statistical significance  Turns out a lot of researchers do not know what p<.05 actually means  Cohen (1994) Article: The earth is round (p<.05)  What it means: "Given that H 0 is true, what is the probability of these (or more extreme) data?”  Trouble is most people want to know "Given these data, what is the probability that H 0 is true?"

Always a difference  Commonly define the null hypothesis as ‘no difference’  Differences between groups always exist (at some level of precision)  Obtaining statistical significance can then be seen as just a matter of sample size  Furthermore the importance and magnitude of an effect are not reflected (because of the role of sample size in probability value attained)

What should we be doing?  Want to make sure we have looked hard enough for the difference – power analysis  Figure out how big the thing we are looking for is – effect size

Calculating effect size  Different statistical tests have different effect sizes developed for them  However, the general principle is the same

Types of effect size Two basic classes of effect size  Focused on means, then standardized differences Allows comparison across samples and variables with differing variance  Equivalent to z scores Note sometimes no need to standardize (units of the scale have inherent meaning)  Variance-accounted-for Amount explained versus the total  Some others (see Kirk, 1996)

Cohen’s d – Differences Between Means  Uuse this with independent samples t test  Getting at: Magnitude of experimental result  Cohen was one of the pioneers  Defined d

The usual problem  In a theoretical discussion use of parameters is fine  We want a practical tool  Use sample statistics Side note - Cohen suggested could use either sample standard deviation, since they should both be roughly equal. In practice people now use the pooled variance.

Characterizing effect size  Cohen emphasized that the interpretation of effects requires the researcher to consider things narrowly in terms of the specific area of inquiry  Evaluation of effect sizes inherently requires a personal value judgment regarding the practical or clinical importance of the effects

  2 =  A measure of the degree to which variability among observations can be attributed to conditions Eta-squared

Small, medium, large? Cohen (1969)  ‘small’ real, but difficult to detect difference between the heights of 15 year old and 16 year old girls in the US  ‘medium’ ‘large enough to be visible to the naked eye’ difference between the heights of 14 & 18 year old girls  ‘large’ ‘grossly perceptible and therefore large’ difference between the heights of 8 & 18 year old girls

How big?  Cohen (e.g. 1969, 1988) offers some rules of thumb Fairly widespread convention now  Looked at social science literature and suggested some ways to carve results into small, medium, and large effects  In terms of d or In terms of  2  0.2 small.10+ large  0.5 medium  0.8 large  Be wary of “mindlessly invoking” these (or any other) criteria, especially for small samples

Maximum Power!  In statistics we want to give ourselves the best chance to find a significant result if one exists.  Power represents the probability of finding that significant result if it exists  Power = 1 -    is our Type II error rate, or the probability of retaining the null when we should have rejected it

Two kinds of power analysis  A priori Used when planning your study What sample size is needed?  Post hoc Used when evaluating study What chance did you have of significant results? Not really appropriate. If you do the power analysis and conduct your analysis accordingly then you did what you could. To say after, “I would have found a difference but didn’t have enough power” isn’t going to impress anyone.

A priori power Can use all this to calculate how many subjects we need to run  Decide an acceptable level of power  Have a standard  level  Figure out the desirable or expected effect size  Calculate N

A priori Effect Size?  Figure out an effect size before I run my experiment?  Several ways to do this: Base it on substantive knowledge  What you know about the situation and scale of measurement Base it on previous research Use Cohen’s conventions

An acceptable level of power?  Why not set power at.99?  Practicalities Cost of increasing power (usually done through increasing n) can be high

Post hoc power  If you fail to reject the null hypothesis you might want to know what chance you had of finding a significant result – defending the failure  This is a little dubious  Better used in a post hoc fashion to figure out the likelihood of other experiments replicating your results

Carrying out the calculation  When you actually have to implement power calculations you can use specialist programs – lots of websites with applications to do the calculation for you

The hard way  Use sample mean (or difference score) and the raw score associated w/ your alpha cutoff point (i.e. convert the t critical value to a raw score).  Use z score methods to help you find percentages for beta and/or 1-beta.

Reminder: Factors affecting Power  Alpha level  Sample size  Effect size  Variability

Howell’s general rule  Look for big effects or  Use big samples