Presentation is loading. Please wait.

Presentation is loading. Please wait.

Does one size really fit all? Evaluating classifiers in Bag-of-Visual-Words classification Christian Hentschel, Harald Sack Hasso Plattner Institute.

Similar presentations


Presentation on theme: "Does one size really fit all? Evaluating classifiers in Bag-of-Visual-Words classification Christian Hentschel, Harald Sack Hasso Plattner Institute."— Presentation transcript:

1 Does one size really fit all? Evaluating classifiers in Bag-of-Visual-Words classification Christian Hentschel, Harald Sack Hasso Plattner Institute

2 Agenda 1.Content-based Image Classification – Motivation 2.Bag-of-Visual-Words 3.Bag-of-Visual-Words Classification ■ Classifier Evaluation ■ Model Visualization 4.Conclusion

3 Content-based Image Classification Christian Hentschel, Does one size really fit all? Chart 3

4 Training: ■ Positive images: (that depict a concept) ■ Negative images: (that don’t) Classification: ■ Test image if it depicts concept (or not): Content-based Image Classification (2) Christian Hentschel, Does one size really fit all? Chart 4

5 ■ Origin - text classification □ e.g. Task: classify forum posts into “insult” (positive) and “not insult” (negative) Bag-of-Visual-Words Christian Hentschel, Does one size really fit all? Chart 5 "haha... at least get your insults straight you idiot!!...." "You're one of my favorite commenter s." { “idiot”: 1, “favorite”: 2, “to”: 3, “you”: 4, “at”: 5, “least”: 6, “commenter”: 7, … } [1, 2, 1, 1, 2, 0, 0,…] [1, 1, 1, 1, 0, 1, 1,…] D1D2 D1 D2

6 ■ Learn a decision rule (e.g. linear SVM) □ i.e. learn features weights Bag-of-Visual-Words (2) Christian Hentschel, Does one size really fit all? Chart 6 [Adopted from A. Mueller, https://github.com/amueller/ml-berlin-tutorial] Feature weights

7 ■ Examples for Visual Words Bag-of-Visual-Words (3) Christian Hentschel, Does one size really fit all? Chart 7 [Schmid, 2013]

8 Bag-of-Visual-Words (4) Christian Hentschel, Does one size really fit all? Chart 8

9 ■ De-facto standard: kernel-based Support Vector Machines □ Decision rule: □ Kernel-Function: □ Distance metric: Bag-of-Visual Words Classification Christian Hentschel, Does one size really fit all? Chart 9

10 ■ Testing different classification models □ Average Precision (AP, area under Precision Recall Curve) ■ Test Dataset □ Caltech-101 – object classes – 31 – 800 images per class ■ Tested Classifiers: □ Naïve Bayes, K-NN, Logistic Regression □ SVM: linear SVM, RBF kernel SVM, Chi 2 -kernel SVM □ Ensemble Methods:Random Forest, AdaBoost □ Hyper parameters optimized in grid-search using CV Bag-of-Visual Words Classification (2) Christian Hentschel, Does one size really fit all? Chart 10

11 ■ Mean AP scores over all classes: Bag-of-Visual Words Classification – Results Christian Hentschel, Does one size really fit all? Chart 11

12 ■ Mean AP scores over all classes: Bag-of-Visual Words Classification – Results Christian Hentschel, Does one size really fit all? Chart 12

13 ■ mAP-scores between best (Chi 2 -SVM) and worst (Naïve Bayes): 0.19 □ Poor performance of Naïve Bayes and k-NN – but fast training ■ Superior performance of kernel-based SVM, but: □ Kernel function (Chi 2 vs. Gaussian RBF) is crucial: – Ensemble methods outperform Gaussian RBF – Gaussian RBF only slightly better than linear SVM □ increased evaluation time: – complex kernel function between each SV and a testing example – ensemble method reduce classification time Bag-of-Visual Words Classification – Results (2) Christian Hentschel, Does one size really fit all? Chart 13

14 ■ Correlation between training sets size and average Precision: Bag-of-Visual Words Classification – Results (3) Christian Hentschel, Does one size really fit all? Chart 14

15 ■ Outliers: □ “minaret” □ “leopards” Bag-of-Visual Words Classification – Results (4) Christian Hentschel, Does one size really fit all? Chart 15

16 ■ Visualize impact of individual image regions on classification result □ Use ensemble methods – No kernel function – AdaBoost: direct indicator for feature importance: mean decrease in impurity Bag-of-Visual Words Classification – Model Visualization Christian Hentschel, Does one size really fit all? Chart 16 Local Region Descriptor BoVW Vector Feature Weights

17 “minaret” Christian Hentschel, Does one size really fit all? Chart 17 ■ “leopards”

18 Christian Hentschel, Does one size really fit all? Chart 18 ■ “minaret”

19 Christian Hentschel, Does one size really fit all? Chart 19 ■ “car_side”

20 Christian Hentschel, Does one size really fit all? Chart 20 ■ “watch”

21 ■ Kernel-based SVM are best choice when aiming for accuracy □ Kernel function is crucial □ Evaluation time-cost is high ■ Ensemble methods are second-best winner □ Fast evaluation □ Offer intuitive visualization of model parameters ■ Visual analytics reveal deficiencies in datasets □ Improperly chosen training data affects classification results Conclusion Christian Hentschel, Does one size really fit all? Chart 21

22 Thank you for your attention! Christian Hentschel, Harald Sack Hasso Plattner Institute


Download ppt "Does one size really fit all? Evaluating classifiers in Bag-of-Visual-Words classification Christian Hentschel, Harald Sack Hasso Plattner Institute."

Similar presentations


Ads by Google