PSY2004 Research Methods PSY2005 Applied Research Methods Week Eleven Stephen Nunn

What it is Why it is important

sensitivity of a statistical test

why stats? variability in the data lots of different, random sources of variability

we’re trying to see if changes in the Independent Variable type of non-word treatment type affects scores on the Dependent Variable reaction time no. of days drugs taken

lots of other things affect the DV individual differences time of day mood level of attention etc etc Lots of random, unsystematic, sources of variation, unrelated to IV ‘noise’

sometimes the effects due to the IV are big, strong easy to see through the noise but what if the effect you’re looking for is small, weak

your ‘equipment’ ( eyes, statistical test) needs to be sensitive enough to spot it otherwise you’ll miss it

sensitivity of a statistical test

ability or probability of detecting an effect [when there is one]

sounds like a good thing [but is often ignored]

Reviews of meta-analyses* suggest most social science effect sizes are medium at best, mostly small (Ellis, 2010) * meta-analyses combine the results from several studies addressing the same hypotheses

Estimates of power in published Psychological research (e.g., Clark-Carter, 1997, looking at BJP) mean power for medium effects = 0.6 mean power for small effects = 0.2 NB recommended level of power = 0.8

What does power = 0.2 mean? [when there is an effect to detect] you only have a 20% chance of detecting it [i.e., getting a statistically significant result]

The ‘noise’ will tend to swamp the effect of your IV. Repeated running of the same study would only give a significant result 20% of the time

Or, you have a 80% probability of making a Type II error [failing to reject the null hypothesis when it is false]

what affects power? anything that changes the effect / ’noise’ ratio

effect size all other things being equal you will have greater power with a bigger effect, less power with a smaller effect

design all other things being equal repeated measures designs are more powerful that independent groups because they allow you to remove the ‘noise’ in the data due to individual differences

cell size all other things being equal simpler designs, fewer levels of your IV will increase power

alpha [criterion for rejecting H 0 ] stricter (smaller) alphas DECREASE power e.g., Post-hoc Type 1 error rate correction Bonferroni achieved at the expense of power

measures, samples unreliable measures heterogeneous samples –> increase the ‘noise’ –> decrease power

sample size a larger N gives you more power [from Central Limit Theorem, increasing N reduces the variability in the sample means, reduces the ‘noise’]

but does this matter?

for the individual researcher: power = 0.2 = highly likely to waste time and other resources

for ‘science’: should we not worry more about Type 1 errors? [rejecting H 0 when it is false]

maybe, but: common (but mistaken) tendency to interpret non-significant results as evidence for no difference i.e., non-significant result due to low power isn’t just waste of resources, but can be misinterpreted in a misleading way

maybe, but: a strong publication bias in Psychology means Type 1 errors and Power are intertwined i.e., only significant results tend to get published

This bias means that if all H 0 were true then all published studies would be Type 1 errors i.e., keeping the type 1 error rate at 5% for individual studies or research as a whole doesn’t keep the error rate in the literature at that level due to the publication bias

Low power across the discipline increases the proportion of published studies that are Type 1 errors i.e., general low power reduces the proportion of studies with false H 0 s that reach significance and which are therefore published (due to the publication bias). The ratio of Type 1 errors to correct rejections of H 0 is therefore increased (Ellis, 2010)

H 0 true (no effect) H 0 true (no effect) H 0 false (effect) H 0 false (effect) Type 1 errors (5%) Correct failure to reject H 0 Correct failure to reject H 0 Correct rejection of H 0 80% power 40% power Type 2 errors Type 1 errors (5%) Type 2 errors Correct rejection of H 0 published ratio of type 1 errors to correct rejections = 5:80 (6.2%) ratio of type 1 errors to correct rejections = 5:40 (12.4%)

NB the publication bias also makes it harder to reveal Type 1 errors in the literature i.e., non-significant failures to replicate a published study (that reveal it as a possible Type 1 error) are less likely to be published due to the publishing bias against non- significant findings.

maybe, but: small sample (i.e., low power) studies tend to over-estimate the size of effects and are more likely to be Type 1 errors (Ellis, 2010) i.e., studies with small N are more likely to give misleading results (but not always)

low power is BAD for individual researchers and BAD for Psychology as a discipline

what you should do : make sure your own study (e.g., FYP) has sufficient power use something like G*Power to calculate your N for power (1-β) = 0.8 simplify your designs, only include what’s necessary an extra IV or condition either reduces power or raises the N you need

what you should do : pay more attention to published studies that have greater power – e.g., larger samples distrust results from small samples look for meta-analyses