Download presentation

1
Boosting Rong Jin

2
**Inefficiency with Bagging**

D Bagging … D1 D2 Dk Boostrap Sampling h1 h2 hk Inefficient boostrap sampling: Every example has equal chance to be sampled No distinction between “easy” examples and “difficult” examples Inefficient model combination: A constant weight for each classifier No distinction between accurate classifiers and inaccurate classifiers

3
**Improve the Efficiency of Bagging**

Better sampling strategy Focus on the examples that are difficult to classify Better combination strategy Accurate model should be assigned larger weights

4
**Intuition + + May overfitting !! Mistakes Classifier3 Classifier2**

No training mistakes !! May overfitting !! + Mistakes X1 Y1 + Mistakes X1 Y1 X3 Y3 Training Examples X1 Y1 X2 Y2 X3 Y3 X4 Y4 Pop up note: no mistakes but possible overfitting on training data Road map before boosting Clearify the old works and the proposed methods Less words !!!

5
AdaBoost Algorithm

6
**AdaBoost Example: t=ln2**

x1, y1 x2, y2 x3, y3 x4, y4 x5, y5 1/5 D0: x5, y5 x3, y3 x1, y1 Sample h1 Training x1, y1 x2, y2 x3, y3 x4, y4 x5, y5 Update Weights h1 Sample x3, y3 x1, y1 2/7 1/7 D1: h2 Training x1, y1 x2, y2 x3, y3 x4, y4 x5, y5 h2 Update Weights 2/9 1/9 4/9 D2: Sample …

7
**How To Choose t in AdaBoost?**

How to construct the best distribution Dt+1(i) Dt+1(i) should be significantly different from Dt(i) Dt+1(i) should create a situation that classifier ht performs poorly

8
**How To Choose t in AdaBoost?**

9
**Optimization View for Choosing t**

ht(x): x{1,-1}; a base (weak) classifier HT(x): a linear combination of basic classifiers Goal: minimize training error Approximate error swith a exponential function

10
**AdaBoost: Greedy Optimization**

Fix HT-1(x), and solve hT(x) and t

11
**Empirical Study of AdaBoost**

AdaBoosting decision trees Generate 50 decision trees by AdaBoost Linearly combine decision trees using the weights of AdaBoost In general: AdaBoost = Bagging > C4.5 AdaBoost usually needs less number of classifiers than Bagging

12
**Bia-Variance Tradeoff for AdaBoost**

AdaBoost can reduce both variance and bias simultaneously variance bias single decision tree Bagging decision tree AdaBoosting decision trees

Similar presentations

OK

A Statistician’s Games * : Bootstrap, Bagging and Boosting * Please refer to “Game theory, on-line prediction and boosting” by Y. Freund and R. Schapire,

A Statistician’s Games * : Bootstrap, Bagging and Boosting * Please refer to “Game theory, on-line prediction and boosting” by Y. Freund and R. Schapire,

© 2018 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on cross site scripting Ppt on trial and error band Ppt on childhood obesity in india Download ppt on non conventional energy resources Seminar ppt on circuit breaker Ppt on small hydro power plant Download ppt on coordinate geometry for class 9th physics Ppt on soil microbial biomass carbon Ppt on advanced construction materials and techniques Spinal cord anatomy and physiology ppt on cells