1. S TATISTICAL S IGNIFICANCE KNR 164

2. W HAT IS S TATISTICAL S IGNIFICANCE ? A statistical hypothesis test is a method of making decisions using data from a scientific study In statistics, a result is called statistically significant if it has been predicted as unlikely to have occurred by chance alone, according to a pre-determined threshold probability, the significance level

3. W HAT IS A P - VALUE ? p -value is the probability that we would have seen our data (or something more unexpected) just by chance if the null hypothesis (null value) is true Small p -values mean the null hypothesis is unlikely given the data

4. W HAT IS A P - VALUE ? Test to see if you can reject the null hypothesis Many common statistical tests, such as chi-squared tests or Student's t-test, produce test statistics which can be interpreted using p -values If you reject the null hypothesis, your results indicate that the observed result would be highly unlikely under the null hypothesis that is, the observation is highly unlikely to be the result of random chance alone

5. W HAT IS A P - VALUE ? Our data are so unlikely given the null hypothesis that I’m going to reject the null hypothesis! Don’t want to reject our data – we want to accept our hypothesis and reject the null as truth (and be confident in doing this)

6. A LPHA Alpha (  ) is the level of significance in hypothesis testing: Alpha is a probability specified before the test is performed. Alpha is the probability of rejecting the null hypothesis when it is true. By convention, typical values of alpha specified in medical research are 0.05 and 0.01.

7. W HAT DOES A LPHA L EVEL M EAN ? Alpha (  ) level indicates at what p -value the null will be rejected Example:  = 0.05; calculated p-value = 0.008; reject null hypothesis  = 0.05; calculated p-value = 0.110; do not reject null hypothesis

8. T HE P - VALUE By convention, p -values of <.05 are often accepted as “statistically significant” in the medical literature; but this is an arbitrary cut-off A cut-off of p <.05 (  =.05) means that in about 5 of 100 experiments, a result would appear significant just by chance

9. H YPOTHESIS T ESTING The Steps: 1. Define your hypotheses (null, alternative) 2. Specify your null distribution 3. Do an experiment 4. Calculate the p -value of what you observed 5. Reject or fail to reject (~accept) the null hypothesis

10. H OW TO INTERPRET A P - VALUE P -value is significant or it is not It is a all-or-none decision Compared to the a priori p-value How confident do you want to be given your result? p <.05 says 95% confident. p =.001 is the same as p =.02 in terms of interpretation as (they are both less then p =.05) Gilbert PB, Berger JO, Stablein D, Becker S, Essex M, Hammer SM, Kim JH, DeGruttola VG. Statistical interpretation of the RV144 HIV vaccine efficacy trial in Thailand: a case study for statistical issues in efficacy trials. J Infect Dis 2011; 203: 969-975.