1. Academic Viva POWER and ERROR T R Wilson

2. Impact Factor Measure reflecting the average number of citations to recent articles published in that journal. For given year, the average number of citations received per paper published from that journal during the two preceding years. E.g. The 2008 impact factor of a journal is Number of times all items published in that journal in 2006 and 2007 were cited by indexed publications during 2008 The total number of "citable items" (not letters or editorials) published by that journal in 2006 and 2007

3. Concept of sampling We are interested in attribute of a population Can’t measure attribute in everyone Therefore we need a representative sample – Random, Unbiased If representative, attribute in sample → estimation of attribute in population Size of sample is important – ↑ sample size →↑accuracy of estimate

4. Statistically speaking Measure attribute in sample – → Sample Mean Estimate of population mean – 95% confidence interval (CI) of sample mean – = 95% chance that the population mean lies within the CI Confidence interval depends on size sample – Larger the sample → the smaller the CI – Smaller the CI → more accurate the estimation

5. Concept statistical testing Interested in difference between an attribute in two populations Take two samples (one from each population) – Measure attribute in each sample – Calculate the difference between the attributes in the two samples – Use this to estimate the difference between the attribute in the two populations – Similarly can produce a statistical estimation of true difference with confidence intervals

6. Concept Null hypothesis From statistical stand point – Assumption no difference between the groups – (Null hypothesis = H 0 ) In the two samples you will almost always find a difference – Is that difference TRUE? (reject Null hypothesis) – Is there really no difference ? (Don’t Reject H 0 ) Need statistical test → P value

7. P Value - definition If you assume that there is no difference in the populations, P value = Probability of finding a difference that large in the samples Small P value – Very unlikely that the observed difference in the samples occurred by chance – Therefore likely to be a difference in populations If P < 0.05 (5%) → difference in sample is unlikely to be coincidental = (statistically) significant

8. P value and Confidence Interval Significant result (p value is <0.05) ≡ 95% CI estimation of population difference will not include 0

9. Error Statistical testing is not infallible Risk of getting a significant result when there is no difference in the populations – False positive (OPTIMISTIC) – Probability of this = TYPE I Error Risk of not getting a significant result when there really is a difference in the populations – False negative (PESSIMISTIC) – Probability of this = TYPE II Error

10. Type I ERROR Even if the P value is small (e.g. p=0.001), there is still a chance that you could have seen the same difference in your samples when there really isn’t any difference in the population (e.g. 1 in 1000) If you took 1000 samples of the two populations with the same mean for testing, one of these 1000 tests would be significant TYPE I ERROR = Level of significance

11. Type II Error TYPE II Error If there is a difference between two populations then 95% CI should exclude 0 If the Sample size is small → the 95%CI for the population difference will be wide and more likely to include 0 giving a false negative result The Type II error can be reduced by increasing the sample size

12. POWER POWER is opposite of TYPE II ERROR TYPE II ERROR = Probability of not getting a significant result when there really is a difference in the population means POWER = Probability of getting a significant result when there is a difference in the population means POWER = 100 – TYPE II ERROR

13. Calculating Power Complicated calculation Requires – Sample size (both samples) – Difference in samples – The spread (standard deviation) of (both) samples – Statistical level of significance of test (usually 5%) ↑Power with – ↑Sample size – ↑Difference – ↑ Level Significance – ↓ Standard deviation

14. Calculating Sample Size Same as equation as for power calculation Requires – Size of difference you want to detect – Estimation of standard deviation in samples – Statistical level of significance of test (usually 5%) – Power that you want test to have (usually 80%) ↑sample size needed for – Small difference – ↑ Standard deviation – ↓ Level Significance (1%) – ↑ Power

15. Beware Errors TYPE I: If a paper does 20 tests at 5% level on two similar populations one of these is likely to be significant TYPE 2: If the study is underpowered and concludes that there is no difference in the main outcome when it looks like there might be (unlikely) TYPE 2: If paper measures a secondary outcome that it is underpowered to detect and concludes that there is no significant difference when it looks like there might be

16. Guide for assessing RCTs