Adaboost and its application training一群classifier的演算法，當我們有待分類物件的database，讓各個classifier去認識這個物件，找出分類的法則，最後再把他們的意見統整起來。 2007.3.2
Outline Introduction Adaboost algorithm Application Two class and training error Multi-class Application Face detection 用例子帶出adaboost這類分類法的概念。
Outline Introduction Adaboost algorithm Application Two class and training error Multi-class Application Face detection
Introduction The horse-track problem: Use historical horse-race data. Odds Dry or muddy Jockey The horse-track problem: Use historical horse-race data. Derive some rules of thumb. Predict the winner. Favorable odds Muddy lightest one 賽馬的賭徒，希望能更有依據的猜測冠軍。收集資料，歸納出一些經驗法則，然後視當天的狀況，套用這些法則，決定要下賭注的馬。
Introduction How to choose horse-race data? Random? Resample data for classifier design. How to combine rules of thumb into single decision? Equal importance? Combine the results of multiple “weak” classifiers into a single “strong” classifier.
Introduction Two most popular: Assume two class to classify : class_1 Bagging ( Breiman,1994 ) Boosting Adaboost ( Freund and Schapire,1996) Assume two class to classify : class_2 class_1
Bagging Input data :x Training data
Boosting Input data :x Training data
Bagging v.s Boosting Bagging: Boosting: Sample: - equal weight. Weak classifier combination : Boosting: -unequal weight. Weak classifier combination: - unequal weight.
Outline Introduction Adaboost algorithm Application Two class and training error Multi-class Application Face detection
A Formal View of Boosting Given training set correct label of instance For Construct distribution on Find weak hypothesis( “rule of thumb”) with small error on Output final hypothesis
Adaboost Concept Adaboost starts with a uniform distribution of “weights” over training examples. The weights tell the learning algorithm the importance of the example. Obtain a weak classifier from the weak learning algorithm, hj(x). Increase the weights on the training examples that were misclassified. (Repeat) At the end, carefully make a linear combination of the weak classifiers obtained at all iterations.
Adaboost Constructing : Final hypothesis: Given and : where = normalized constant, Final hypothesis:
Adaboost (Example)
Adaboost (Example)
Adaboost (Example)
Adaboost (Example)
Adaboost (Example)
Advantages of Adaboost Weight update focus more on “hard” samples. ( misclassified in the previous iterations) Simple and easy to program. No parameter to tune( except T). Can combine with many classifiers to find weak hypothesis: Neural network, decision trees, nearest-neighbor classifiers….. Classifier的分類法不會一直被已經分類正確的sample所影響，例如：橄欖球隊員，壯且高，我們先用體重去分類，把用體重分類正確的那些人 的影響降低，在考慮身高。
Training Error Let , then training error So if then training error Error bound和每一次的錯誤率有關，假設某一個分類器的error很小，可以彌補其他分類不好的分類器。 T越大，越多分類器，理論上training error 可以趨近於零。
Multi-class Problem Adaboost.MH e.g: Reduce to binary problems. Possible labels are {a,b,c,d,e} Each training sample replaced by five {-1,+1} labeled sample.
Adaboost.MH Formally:
Outline Introduction Adaboost algorithm Application Two class and training error Multi-class Application Face detection
Face Detection Non-face Adaboost Detection result Training set
Classifiers Design Haar-like features for : Two-rectangle (A,B) Three-rectangle (C) Four-rectangle (D) 24 24
Classifiers Design Why use Haar-like features? Resolution of detector : 24*24 total 160,000 (quite large)
Classifiers Design Use “Integral image”. Feature computation:
Classifier Design Choose the best features Adaptive reweighting Non-face Training set Haar-like features 自動挑選適合的feature.使我們可以得到更efficient 的feature。
Face Detection Computation cost: Ex: image size: 320x240. sub-window size:24x24. frame rate: 15 frame/sec. each feature need (320-24+1)x(240-24+1)x15=966,735 per sec (if ignore scaling) huge computation cost !!
Face Detection Use cascade classifiers. Example: 200 feature classifier 10 20-featureclassifiers.
Face Detection Advantage of cascade classifiers: Maintain accuracy. Speed up.
Experiments