P value and confidence intervals Imre Janszky Faculty of Medicine NTNU

” The statistical education of scientists emphasizes a flawed approach to data analysis that should have been discarded long ago. This defective method is statistical significance testing. ... It has produced countless misinterpretations of data that are often amusing for their folly, but also hair-raising in view of the serious consequences.” Kenneth Rothman

Overview P value, what is it, what is it not, misconceptions and problems with p value Statistical significance and hypothesis testing, misconceptions and problems with statistical significance History of statistical significance Confidence intervals and estimation

P value and hypothesis testing We need : A sample from a population Set up a null hypothesis concerning the population Set up an alternative hypothesis concerning the population Contrast the data with the null hypothesis

Example – body weight of men and women A sample from the big population of men and women The null hypothesis: women and men have the same body weight, ∆weight = 0 Alternative hypothesis: women and men have a different body weight, ∆weight ≠ 0

P value- What is it? A measure of the consistency between the data obtained in a sample and a null hypothesis concerning the big population A simplified definition: P value usually refers to the probability, assuming that the null hypothesis is true, that the data obtained in a study would demonstrate an association as far from the null-hypothesis as, or farther than, what was actually obtained

Example – body weight of men and women We observed that in our sample, ∆weight=5 kg, i.e., on average, men are heavier than women by 5 kg P=0.01 Using the simplified definition: If in the big population of men and women ∆weight=0, then the probability to obtain a difference of +5 kg or higher than that or to obtain a difference of -5kg or lower than that between men and women in our sample is 1%

P value - What is it not? ≠ the probability that the null hypothesis is true ≠ the probability that a finding is just by chance ≠ strength of the association ≠ degree of uncertainty

Statistical significance, hypothesis testing Based on a predefined cut off for p value Typically, the null hypothesis is rejected if p<0.05 (and the results are called statistically significant) and not rejected if p>0.05 (and called statistically non-significant) Type I error is the incorrect rejection of a true null hypothesis Type II error is the incorrect lack of rejection of a false null hypothesis

Problems with statistical significance Court analogy: the defendant is assumed to be innocent until it is proved otherwise/ the null hypotheses is assumed to be true until we can reject it However, decisions in medicine are not be based on a p value from a single study In contrast with the courts, “gray zones” are acceptable Very common misconception: lack of significance is an evidence for lack of effect Loss of important information with simple dichotomization

History of statistical significance 1. First use of statistical significance (John Venn, 1888): the mean height of 2,315 criminals differs from the mean height of 8,585 members of the general adult population by about two inches. The chance to observe this is very low, many billions to one, if there would be no difference. Therefore, Venn concluded, it is likely that the difference is real, or statistically significant

History of statistical significance 2. Modern use of statistical significance was developed in business applications needing decisions based on single results At the beginning there was no universal agreement on what should be the cut-off, or whether an universal cut-off is needed at all

History of statistical significance 3. Developers of hypothesis testing (Egon Pearson and Jerzy Neyman) emphasized that the cut off values should be chosen based on the concrete situation Roland Fisher published a very influential statistical textbook:Statistical Methods for Research Workers (first edition in 1925)

History of statistical significance 4. In his book, Fisher published helping aids (distribution tables for different test statistics) for the calculation of selected p values, including 0.05 He also stated: ”Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. ”

Why statistical significance become so widespread? Relatively easy calculation (it was easier to claim statistical significance vs. no significance than calculating a concrete p value) Feeling of objectivity Simplified interpretation for the researcher and for the readers Easy writing False impression of certainty

Estimation and CI In contrast to hypothesis testing, we might be interested in estimating a certain population parameter (e.g. ∆weight in the previous example) It is a quantitative rather than qualitative approach We provide the single most likely value (point estimate) and the uncertainty around it (confidence intervals, CI)

Estimation and CI 95% CI= over unlimited repetitions of the study the 95% confidence interval will contain the true population parameter with a frequency of no less than 95% If a 95% CI does not include the value for the null hypothesis (0 for continuous measures, e.g. ∆weight, risk or rate difference, but 1 for relative measures, e.g. risk ratio, incidence rate ratio, relative risk, odds ratio) then the finding is statistically significant

Estimation and CI However, it is much more important that the point estimate and its CI allows you to consider the effect size and its uncertainty separately

Summary P value is often misinterpreted We shall avoid “statistical significance” as a simple dichotomy of study results Use of confidence intervals is preferable over p values