1. P value and confidence intervals
Imre Janszky
Faculty of Medicine
NTNU

2. ” The statistical education of scientists emphasizes a flawed approach to data analysis that should have been discarded long ago. This defective method is statistical significance testing. ... It has produced countless misinterpretations of data that are often amusing for their folly, but also hair-raising in view of the serious consequences.”
Kenneth Rothman

3. Overview
P value, what is it, what is it not, misconceptions and problems with p value
Statistical significance and hypothesis testing, misconceptions and problems with statistical significance
History of statistical significance
Confidence intervals and estimation

4. P value and hypothesis testing
We need :
A sample from a population
Set up a null hypothesis concerning the population
Set up an alternative hypothesis concerning the population
Contrast the data with the null hypothesis

5. Example – body weight of men and women
A sample from the big population of men and women
The null hypothesis: women and men have the same body weight, ∆weight = 0
Alternative hypothesis: women and men have a different body weight, ∆weight ≠ 0

6. P value- What is it?
A measure of the consistency between the data obtained in a sample and a null hypothesis concerning the big population
A simplified definition: P value usually refers to the probability, assuming that the null hypothesis is true, that the data obtained in a study would demonstrate an association as far from the null-hypothesis as, or farther than, what was actually obtained

7. Example – body weight of men and women
We observed that in our sample, ∆weight=5 kg, i.e., on average, men are heavier than women by 5 kg
P=0.01
Using the simplified definition: If in the big population of men and women ∆weight=0, then the probability to obtain a difference of +5 kg or higher than that or to obtain a difference of -5kg or lower than that between men and women in our sample is 1%

8. P value - What is it not?
≠ the probability that the null hypothesis is true
≠ the probability that a finding is just by chance
≠ strength of the association
≠ degree of uncertainty

9. Statistical significance, hypothesis testing
Based on a predefined cut off for p value
Typically, the null hypothesis is rejected if p<0.05 (and the results are called statistically significant) and not rejected if p>0.05 (and called statistically non-significant)
Type I error is the incorrect rejection of a true null hypothesis
Type II error is the incorrect lack of rejection of a false null hypothesis

10. Problems with statistical significance
Court analogy: the defendant is assumed to be innocent until it is proved otherwise/ the null hypotheses is assumed to be true until we can reject it
However, decisions in medicine are not be based on a p value from a single study
In contrast with the courts, “gray zones” are acceptable
Very common misconception: lack of significance is an evidence for lack of effect
Loss of important information with simple dichotomization

11. History of statistical significance 1.
First use of statistical significance (John Venn, 1888): the mean height of 2,315 criminals differs from the mean height of 8,585 members of the general adult population by about two inches. The chance to observe this is very low, many billions to one, if there would be no difference. Therefore, Venn concluded, it is likely that the difference is real, or statistically significant

12. History of statistical significance 2.
Modern use of statistical significance was developed in business applications needing decisions based on single results
At the beginning there was no universal agreement on what should be the cut-off, or whether an universal cut-off is needed at all

13. History of statistical significance 3.
Developers of hypothesis testing (Egon Pearson and Jerzy Neyman) emphasized that the cut off values should be chosen based on the concrete situation
Roland Fisher published a very influential statistical textbook:Statistical Methods for Research Workers (first edition in 1925)

14. History of statistical significance 4.
In his book, Fisher published helping aids (distribution tables for different test statistics) for the calculation of selected p values, including 0.05
He also stated: ”Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. ”

15. Why statistical significance become so widespread?
Relatively easy calculation (it was easier to claim statistical significance vs. no significance than calculating a concrete p value)
Feeling of objectivity
Simplified interpretation for the researcher and for the readers
Easy writing
False impression of certainty

16. Estimation and CI
In contrast to hypothesis testing, we might be interested in estimating a certain population parameter (e.g. ∆weight in the previous example)
It is a quantitative rather than qualitative approach
We provide the single most likely value (point estimate) and the uncertainty around it (confidence intervals, CI)

17. Estimation and CI
95% CI= over unlimited repetitions of the study the 95% confidence interval will contain the true population parameter with a frequency of no less than 95%
If a 95% CI does not include the value for the null hypothesis (0 for continuous measures, e.g. ∆weight, risk or rate difference, but 1 for relative measures, e.g. risk ratio, incidence rate ratio, relative risk, odds ratio) then the finding is statistically significant

18. Estimation and CI
However, it is much more important that the point estimate and its CI allows you to consider the effect size and its uncertainty separately

19. Summary P value is often misinterpreted
We shall avoid “statistical significance” as a simple dichotomy of study results
Use of confidence intervals is preferable over p values