1. End of Chapter 8
Neil Weisenfeld
March 28, 2005

2. Outline 8.6 MCMC for Sampling from the Posterior 8.7 Bagging
8.7.1 Examples: Trees with Simulated Data
8.8 Model Averaging and Stacking
8.9 Stochastic Search: Bumping

3. MCMC for Sampling from the Posterior
Markov chain Monte Carlo method
Estimate parameters given a Bayesian model and sampling from the posterior distribution
Gibbs sampling, a form of MCMC, is like EM only sample from conditional dist rather than maximizing

4. Gibbs Sampling Wish to draw a sample from the joint distribution
If this is difficult, but it’s easy to simulate conditional distributions
Gibbs sampler simulates each of these
Process produces a Markov chain with stationary distribution equal to desired joint disttribution

5. Algorithm 8.3: Gibbs Sampler
Take some initial values
for t=1,2,…:
for k=1,2,…,K generate from:
Continue step 2 until joint distribution of
does not change

6. Gibbs Sampling
Only need to be able to sample from conditional distribution, but if it is known, then:
is a better estimate

7. Gibbs sampling for mixtures
Consider latent data from EM procedure to be another parameter:
See algorithm (next slide), same as EM except sample instead of maximize
Additional steps can be added to include other informative priors

8. Algorithm 8.4: Gibbs sampling for mixtures
Take some initial values
Repeat for t=1,2,…,
For I=1,2,…,N generate
Set
Continue step 2 until the joint distribution of
doesn’t change.

9. Figure 8.8: Gibbs Sampling from Mixtures
Simplified case with fixed variances and mixing proportion

10. Outline 8.6 MCMC for Sampling from the Posterior 8.7 Bagging
8.7.1 Examples: Trees with Simulated Data
8.8 Model Averaging and Stacking
8.9 Stochastic Search: Bumping

11. 8.7 Bagging Using bootstrap to improve the estimate itself
Bootstrap mean approximately posterior average
Consider regression problem:
Bagging averages estimates over bootstrap samples to produce:

12. Bagging, cnt’d
Point is to reduce variance of the estimate while leaving bias unchanged
Monte-Carlo estimate of “true” bagging estimate, approaching as
Bagged estimate will differ from the original estimate only when latter is adaptive or non-linear function of the data

13. Bagging B-Spline Example
Bagging would average the curves in the lower left-hand corner at each x value.

14. Quick Tree Intro
Can’t do.
Recursive subdivision.
Tree.
f-hat.

15. Spam Example

16. Bagging Trees Each run produces different trees
Each tree may have different terminal nodes
Bagged estimate is the average prediction at x from the B trees. Prediction can be a 0/1 indicator function, in which case bagging gives a pk proportion of trees predicting class k at x.

17. 8.7.1: Example Trees with Simulated Data
Original and 5 bootstrap-grown trees
Two classes, five features, Gaussian distribution
Y from
Bayes error 0.2
Trees fit to 200 bootstrap samples

18. Example Performance
High variance among trees because features have pairwise correlation 0.95.
Bagging successfully smooths out vairance and reduces test error.

19. Where Bagging Doesn’t Help
Classifier is a single axis-oriented split.
Split is chosen along either x1 or x2 in order to minimize training error.
Boosting is shown on the right.

20. Outline 8.6 MCMC for Sampling from the Posterior 8.7 Bagging
8.7.1 Examples: Trees with Simulated Data
8.8 Model Averaging and Stacking
8.9 Stochastic Search: Bumping

21. Model Averaging and Stacking
More general Bayesian model averaging
Given candidate models Mm, m =1…M and a training set Z and
Bayesian prediction is weighted avg of indiv predictions with weights proportional to posterior of each model

22. Other Averaging Strategies
Simple unweighted average of predictions (each model equally likely)
BIC: use to estimate posterior model probabilities: weight each model depending on fit and how many parameters it uses
Full Bayesian strategy:

23. Frequentist Viewpoint of Averaging
Given a set of predictions from M models, we seek optimal weights w:
Input x is fixed and N observations in Z are distributed according to P. Solution is the linear regression of Y on the vector of model predictions:

24. Notes of Frequentist Viewpoint
At the population level, adding models with arbitrary weights can only help.
But the population is, of course, not available
Regression over training set can be used, but this may not be ideal: model complexity not taken into account…

25. Stacked Generalization, Stacking
Cross validated predictions avoid unfairly high weight to models with high complexity
If w restricted to vectors with one unit weight and the rest zero, model choice has smallest leave-one-out cross validation
In practice we use combined models with optimal weights: better prediction, but less interpretability

26. Outline 8.6 MCMC for Sampling from the Posterior 8.7 Bagging
8.7.1 Examples: Trees with Simulated Data
8.8 Model Averaging and Stacking
8.9 Stochastic Search: Bumping

27. Stochastic Search: Bumping
Rather than average models, try to find a better single model.
Good for avoiding local minima in the fitting method.
Like bagging, draw bootstrap samples and fit model to each, but choose model that best fits the training data

28. Stochastic Search: Bumping
Given B bootstrap samples Z*1,…, Z*B, fitting model to each yields predictions:
For squared error, choose model from bootstrap sample:
Bumping tries to move around the model space by perturbing the data.

29. A contrived case where bumping helps
Greedy tree-based algorithm tries to split on each dimension separately, first one, then the other.
Bumping stumbles upon the right answer.