Introduction to hypothesis testing

Idea 1. Formulate research hypothesis H 1 New theory, effect of a treatment etc. 2. Formulate an opposite hypothesis H 0 Theory is wrong, there is no effect, status quo This is often called “null hypothesis” The two hypotheses should be mutually exclusive: one or the other must be true 3. See if H 0 could be rejected

Rules for rejecting Assume that H 0 is true Use existing knowledge to construct a probability model for hypothetical data: what would be the relative frequency distribution of hypothetical data x after infinite repetition of sampling if the H 0 was true i.e. define p(x | H 0 ) Set up a decision rule: if the actual observed data x 0 hits a “critical region” then reject H 0 accept H 1

Choosing the critical region There is no theory for that Common practice: P(x>k |H 0 )=0.05, k is called “critical value” If x 0 >= k, then reject H 0 If x 0 <k, do not reject H 0 In other words: if P(x>x 0 | H 0 )<0.05, then reject H 0 P(x>x 0 |H 0 ) : p-value

P-value in words Frequency probability of hypothetical data being more “extreme” than the observed data we have, given that the null hypothesis is true. Frequency probability statement about data that we do not have under the assumption that null hypothesis is true

What p-value is NOT 1. Probability of hypothesis being true given the observed data P(H 0 |x 0 ) 2. Probability or probability density of observed data given that the hypothesis is true P(x 0 |H 0 ) 3. Risk of being wrong if you claim that H 0 is false

How to interpret in terms of H 0 ? There is no quantitative interpretation Qualitatively, what can it mean: If p-value <0.05 : H 0 is false, there is a true and practically meaningful effect H 0 is false, there is a true but practically meaningless effect, your sample size was large enough to pick that up H 0 is true, you were just “lucky” If p-value >0.05 H 0 is false, there is a meaningful effect, but your sample size was too small to pick that up H 0 is false, there is a tiny effect, but your sample size was too small to pick that up H 0 is true, you were not lucky Effect size, amount of data and plausibility of H 0 all affect the qualitative conclusion

Statistical significance? If the test statistics hits the critical region, the observed effect is often said to be statistically significant However: there is no statistical theory which would define the concept directly Meaning of statistical significance is defined by the person who designs the test: choice of critical region

Publication bias H 1 : Rats who hear Rolling Stones live longer or shorter than those who hear Led Zeppelin H 0 : No effect 100 independent experiments by different research teams: 8 studies with p-value < 0.05, 92 studies with p-value > 0.05 Because journals like p-value <0.05, those papers are much more likely to be published

Consequence? When studied enough, p-value will eventually be less than 0.05 independent of the true state of H 0 Sensational findings pop-up easily Publication policy exaggerates the phenomenon Interpretation of p-value, once again? small p-value: there is a potential mismatch between the observations and H 0 not-so-small p-value: H 0 predicts the data reasonably well Note, regardless of the p-value, the “mismatch” between H 1 and data could be small or large

Example: effect of a treatment H 0 : no effect: the means are equal H 1 : some effect: the means differ Assume equal and known variance in both groups Assume normal distribution in both groups Data: 10 measurements from both control and treatment Summarize data with a statistic: m: group mean s: group standard deviation

Distribution of hypothetical data Under H 0, the t-statistic will follow a t- distribution with degrees of freedom equal to 18 (number of observations -2) when sampling is repeated infinitely Critical region defined by the t-distribution after choosing 0.05 as significance level P(| t |>k | H 0 )=0.05, k=2.1 if |t| > 2.1, reject H 0