Presentation is loading. Please wait.

Presentation is loading. Please wait.

Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficient boostrap sampling: Every example.

Similar presentations


Presentation on theme: "Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficient boostrap sampling: Every example."— Presentation transcript:

1 Boosting Rong Jin

2 Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficient boostrap sampling: Every example has equal chance to be sampled No distinction between “easy” examples and “difficult” examples Inefficient model combination: A constant weight for each classifier No distinction between accurate classifiers and inaccurate classifiers

3 Improve the Efficiency of Bagging Better sampling strategy Focus on the examples that are difficult to classify Better combination strategy Accurate model should be assigned larger weights

4 Intuition Training Examples X1Y1X1Y1 X2Y2X2Y2 X3Y3X3Y3 X4Y4X4Y4 Mistakes X1Y1X1Y1 X3Y3X3Y3 Classifier1 Classifier2 Mistakes X1Y1X1Y1 + Classifier3  No training mistakes !!  May overfitting !! +

5 AdaBoost Algorithm

6 AdaBoost Example:  t =ln2 x 1, y 1 x 2, y 2 x 3, y 3 x 4, y 4 x 5, y 5 1/5 D0:D0: x 5, y 5 x 3, y 3 x 1, y 1 Sample h1h1 Training 2/71/72/7 1/7 D1:D1: x 1, y 1 x 2, y 2 x 3, y 3 x 4, y 4 x 5, y 5 Update Weights h1h1   Sample x 3, y 3 x 1, y 1 h2h2 Training x 1, y 1 x 2, y 2 x 3, y 3 x 4, y 4 x 5, y 5    h2h2 Update Weights 2/91/94/9 1/9 D2:D2: Sample …

7 How To Choose  t in AdaBoost? How to construct the best distribution D t+1 (i) 1.D t+1 (i) should be significantly different from D t (i) 2.D t+1 (i) should create a situation that classifier h t performs poorly

8 How To Choose  t in AdaBoost?

9 Optimization View for Choosing  t h t (x): x  {1,-1}; a base (weak) classifier H T (x): a linear combination of basic classifiers Goal: minimize training error Approximate error swith a exponential function

10 AdaBoost: Greedy Optimization Fix H T-1 (x), and solve h T (x) and  t

11 Empirical Study of AdaBoost AdaBoosting decision trees Generate 50 decision trees by AdaBoost Linearly combine decision trees using the weights of AdaBoost In general: AdaBoost = Bagging > C4.5 AdaBoost usually needs less number of classifiers than Bagging

12 Bia-Variance Tradeoff for AdaBoost AdaBoost can reduce both variance and bias simultaneously single decision tree Bagging decision tree bias variance AdaBoosting decision trees


Download ppt "Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficient boostrap sampling: Every example."

Similar presentations


Ads by Google