2 Main points Definition of confidence intervals Definition of p-value Why use bootstrap, why use permutation tests?What are the differences between bootstrap and permutation tests?How to use bootstrap?How to use permutation tests?
3 Definition of confidence intervals A confidence interval represents the precision of the estimation of a test statisticIf the same experiment was replicated a hundred times, the 95% CI would, on average, contain the estimated TS in 95 of these samples.
4 Definition of p-valueA p-value represents the probability of observing a TS this extreme or more extreme if the null hypothesis is true
5 Why use bootstrap, why use permutation tests? Test statisticTheoretical hypothesisOperational hypothesisNull hypothesisP-valueORCIUnderlying distributionIf one of those is unusual or unknown, bootstrap or permutation is useful5
6 Test statistics Is not implemented in standard software Is estimated by a single value in the whole sampleIs a relationship between two TS
7 Null hypothesisIs different from zero in a way that is hard to quantifyExample: If you are expecting a certain variable to explain more than 50 percent of a different variable, such that the r squared is greater than 50. We would want to bootstrap the r-statistics and see whether the confidence interval includes 50, or values below 50. If the confidence interval does not, then we can say that our hypothesis is supported. If it does, then we cannot reject the null.
8 Underlying distribution Is unknown because TS was unknownIs unknown because conditions of applications for parametric tests do not seem to be metIs known for the null hypothesis but seems likely to be different for the alternative hypothesis p-value should be correct but CI will be incorrect.
9 Differences between bootstrap and permutation tests estimates confidence interval, bias and standard errorSimulates data under the alternative hypothesisSampling is done with replacement of subjectsMany bootstrap samples because of replacementPermutation testsestimates p-value and distribution under the null.Simulates data under the null hypothesisSampling is done without replacement of subjectsFinite number of potential permutation samples
10 Main points: how to How to bootstrap a test statistics Determine the test statistic of interest (must be a single value)What is randomly sampled? How many subjects in the bootstrap samples (with or without replacement)? How many bootstrap samplesExamine histogram of TS* with TS (the observed TS), average of TS*, and boundaries of percentile and bca CIInterpret results
11 Main points: how to How to use a permutation test Determine the test statistic of interest (must be a single value)What must be shuffled in order to simulate what happens under the null hypothesis? How many subjects in the permutation samples (with or without replacement)? How many permutation samples?Examine histogram of TS* with TS (the observed TS)Compute 2 p-values if possible: one-tailed, two-tailedInterpret results
12 Extra assignment Read the article by Zentner et al. (2007). If an hypothesis is tested by either bootstrap or permutation test, describe in details:the hypothesis,How the hypothesis was operationalizedThe procedure,The resultsThe interpretation of the results.