Download presentation

Presentation is loading. Please wait.

Published byJourney Wheaton Modified about 1 year ago

1
A Statistician’s Games * : Bootstrap, Bagging and Boosting * Please refer to “Game theory, on-line prediction and boosting” by Y. Freund and R. Schapire, Proceedings of 9th Conference on Computational Learning Theory. Yaochu Jin Future Technology Research Honda R&D Europe (Germany) March 21, 2000

2
Bootstrap -- Problem Description The bootstrap was introduced as a general method for assessing the statistical accuracy of an estimator Given data: x = ( x 1,..., x n ) Have an estimator: = s(x) ? How to assess the accuracy of

3
Bootstrap -- the Idea Bootstrap estimate of the standard error:

4
Bootstrap -- Pros and Cons Easy to implement Need a large number of independent bootstrap samples (B>=1000) Uncertainty of the estimate 1) Jackknife-after-Bootstrap(JAB) 2) Weighted JAB

5
Bagging is Not Related to Begging Using bootstrap techniques to improve the estimator Bagging -- Bootstrap aggregating

6
Bagging -- the Idea The final estimate: = ( 1 + 2 +... + B )/B

7
Bagging -- Pros and Cons The estimator can be significantly improved if the learning algorithm is unstable Degrade the performance of stable procedures Reduce the variance, bias unchanged

8
Adaptive Bagging Reduce both variance and bias

9
Boosting To boost a “weak” learning algorithm into a “strong” learning algorithm A week learning algorithm can be inaccurate rules of thumb that is slightly better than random guess

10
AdaBoost Initialize Distribution D 1 (i) = 1/n Calculate error t Choose weight t = 1/2ln(1- t / t ) Update distribution The final estimate: = ( 1 1 + 2 2 +... + n B )/B

11
AdaBoost -- Pros and Cons Reduce both variance and bias Need large number of estimators (B>=1000) Sensitive to noise Theoretical guarantee (maximizes the likelihood) Easy to implement (compared to Bayesian methods) Relation to Support Vector Machines

12
Further Information on B 3 http://www-stat.stanford.edu/~tibs/ ftp://ftp.stat.berkeley.edu/pub/users/breiman/ http://www.research.att.com/~yoav/ http://www.research.att.com/~schapire/

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google