Introduction to resampling techniques for generating confidence measures

Resampling techniques Randomization Resampling without replacement (re-ordering, permutations) Jackknife Leaving one data point out at a time (not good for small sample sizes), in paleobiology usually used for phylogenetic analyses Sampling Standardization When comparing samples of different sizes Bootstrap Parametric Generate datasets from a parametrized model and comparing these with empirical data Non parametric Most common in paleobiology

Randomization Empirical Data Randomized Sample 1 Randomized Sample 2 …. Randomized Sample N

Jack-Knife Empirical Data Jack knife sample 3 Jack knife sample 1 …..Jack knife sample N

Sampling Standardization Empirical data 1 Empirical data 2 Empirical data 3 Standardized Sample 1 Standardized Sample 2 … Standardized Sample N

Non-parametric bootstrap Empirical Data Non-parametric bootstrap Bootstrapped Sample 3 Bootstrapped Sample 1 Bootstrapped Sample 2 ….. Bootstrapped Sample N

Non-parametric bootstrap Parametric bootstrap Empirical data Bootstraps samples Empirical data Simulated samples Estimate parameters (model) Estimate parameters Estimate parameters (model) Estimate parameters

Resampling techniques Randomization Resampling without replacement (re-ordering, permutations) Jackknife Leaving one data point out at a time (not good for small sample sizes), in paleobiology usually used for phylogenetic analyses Sampling Standardization When comparing samples of different sizes Bootstrap Parametric Generate datasets from a parametrized model and comparing these with empirical data Non parametric Most common in paleobiology

Why resampling (now) Underlying distribution of data not well understood and/or complex Convenient way to generate uncertainty measures Computer intensive (possible only with faster computers)

Bootstrapping construct estimate of frequency distributions expected from a “generative process” Equivalent to generating replicate outcomes from an experiment (doing something many times to see the range of results) Assumption: data are representative sample of independent observations derived randomly from the studied statistical population

Bootstrap error estimates Estimate standard error by resampling from the single sample we have. This approach uses sampling with replacement from observed sample to simulate sampling without replacement from the underlying distribution. Procedure Start with observed sample of size n and observed sample statistic, call it Z. Randomly pick a sample of size n, with replacement, from the observedsample. Calculate the sample statistic of interest on this random sample; call isZboot. Repeat many times (generally hundreds to thousands, ideally untilestimate of SE stabilizes). Calculate standard deviation of the Zboot. This is an estimate of the standard error of the observed sample statistic Z:SD(Zboot) ≈ SE(Z)

Example (sampling standardization) Alroy et al. 2008. Phanerozoic trends in the global diversity of marine invertebrates. Science 321:97-100

Example (non parametric bootstrap) Foote, M. 2006. Substrate affinity and diversity dynamics of Paleozoic marine animals Paleobiology 32:345-366.

Example (non parametric bootstrap) Liow et al- 2009. Lower extinction risk in Sleep-or-Hide Mammals. Am Nat 173:264–272.

R demo Packages (e.g. boot, boostrap) Write your own: use the function sample Nice help http://www.ats.ucla.edu/stat/r/library/bootstrap.htm

Links http://www.paleo.geos.vt.edu/MK/Kowalewski_PNG_2010.pdf http://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf