Download presentation

Presentation is loading. Please wait.

Published byZavier Donson Modified about 1 year ago

1
Boosting Rong Jin

2
Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficient boostrap sampling: Every example has equal chance to be sampled No distinction between “easy” examples and “difficult” examples Inefficient model combination: A constant weight for each classifier No distinction between accurate classifiers and inaccurate classifiers

3
Improve the Efficiency of Bagging Better sampling strategy Focus on the examples that are difficult to classify Better combination strategy Accurate model should be assigned larger weights

4
Intuition Training Examples X1Y1X1Y1 X2Y2X2Y2 X3Y3X3Y3 X4Y4X4Y4 Mistakes X1Y1X1Y1 X3Y3X3Y3 Classifier1 Classifier2 Mistakes X1Y1X1Y1 + Classifier3 No training mistakes !! May overfitting !! +

5
AdaBoost Algorithm

6
AdaBoost Example: t =ln2 x 1, y 1 x 2, y 2 x 3, y 3 x 4, y 4 x 5, y 5 1/5 D0:D0: x 5, y 5 x 3, y 3 x 1, y 1 Sample h1h1 Training 2/71/72/7 1/7 D1:D1: x 1, y 1 x 2, y 2 x 3, y 3 x 4, y 4 x 5, y 5 Update Weights h1h1 Sample x 3, y 3 x 1, y 1 h2h2 Training x 1, y 1 x 2, y 2 x 3, y 3 x 4, y 4 x 5, y 5 h2h2 Update Weights 2/91/94/9 1/9 D2:D2: Sample …

7
How To Choose t in AdaBoost? How to construct the best distribution D t+1 (i) 1.D t+1 (i) should be significantly different from D t (i) 2.D t+1 (i) should create a situation that classifier h t performs poorly

8
How To Choose t in AdaBoost?

9
Optimization View for Choosing t h t (x): x {1,-1}; a base (weak) classifier H T (x): a linear combination of basic classifiers Goal: minimize training error Approximate error swith a exponential function

10
AdaBoost: Greedy Optimization Fix H T-1 (x), and solve h T (x) and t

11
Empirical Study of AdaBoost AdaBoosting decision trees Generate 50 decision trees by AdaBoost Linearly combine decision trees using the weights of AdaBoost In general: AdaBoost = Bagging > C4.5 AdaBoost usually needs less number of classifiers than Bagging

12
Bia-Variance Tradeoff for AdaBoost AdaBoost can reduce both variance and bias simultaneously single decision tree Bagging decision tree bias variance AdaBoosting decision trees

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google