1. Semantic Kernel Forests from Multiple Taxonomies Sung Ju Hwang (University of Texas at Austin), Fei Sha (University of Southern California), and Kristen Grauman (University of Texas at Austin)

2. Limitation of status quo recognition Until recently, most categorization methods solely relied on the category labels, treating each instance as an isolated entity. Semantic space Visual world Cat Dog Wolf Zebra 1 2 3 4 x x x x

3. Limitation of status quo recognition Semantic space Cat Wolf Zebra Canine Visual world Dog Pet Wild Similar Dissimilar However, semantic entities exist in relation to others. Larger and finer-grained datasets → more meaningful relations How can we exploit such relations for improved categorization? [Fergus10] Semantic Label Sharing for Learning with Many Categories, R. Fergus, H. Bernal, Y. Weiss, A. Torralba,, ECCV 2010 [Zhao11] Large Scale Category Structure Aware Image Classification, B. Zhao, L. Fei Fei, E. P. Xing, NIPS 2011

4. Motivation Our focus: a semantic taxonomy 1) Partial alignment between the taxonomy and visual distribution DalmatianWolf Siam. Cat Domestic leopard Wild Tameness DalmatianWolfLeopard Spotted Siam. Cat Pointy Corner Texture Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Biological Appearance Habitat 2) No single ‘optimal’ taxonomy - But, potentially two snags. What information to exploit from multiple taxonomies and how to leverage it?

5. Idea DalmatianWolf Siam. Cat Domestic leopard Wild Tameness Exploit multiple semantic taxonomies for visual feature learning DalmatianWolfLeopard Spotted Siam. Cat Pointy Corner Texture Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Dog-like shape Cat-face Biological Appearance Habitat SpotPointy corner Indoor setting, person Woods - Taxonomies provide human merge/split criteria - Each taxonomy provides complementary information How do we then, 1. Learn granularity and view specific features on each taxonomy, and 2. Combine learned features across taxonomies for object recognition?

6. Overview Goal: Learn and combine features across multiple taxonomies DalmatianWolf Siam. Cat Domestic leopard Wild Tameness DalmatianWolfLeopard Spotted Siam. Cat Pointy Corner Texture Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Dog-like shape Cat-face SpotPointy corner Indoor setting, person Woods Dog-like shape Cat-face Spot Pointy corner Indoor setting, person Woods 1. Learn view and granularity specific features at each taxonomy Categorization model 2. Optimally combine learned features for categorization +++ ++ [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011

7. Tree of Metrics How to learn granularity- and view- specific features? Siamese cat Persian cat Canine Carnivore Feline Domestic Cat Dalmatian WolfBit cat Intuition: Features useful for the discrimination of the superclasses less useful for subcategory discrimination – Exploit parent-child relationship to isolate features at each node [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011

8. Given a taxonomy,we learn a metric for each internal (superclass) node n to discriminate between its subclasses. Tree of Metrics Approach the feature learning problem as hierarchical metric learning with disjoint regularization xixi xjxj xlxl Feline Canine margin [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011 Lighter element has higher value Siamese cat Persian cat Carnivore Domestic Cat Wolf CanineFeline Dalmatian Big cat

9. Tree of Metrics Siamese cat Persian cat Carnivore Domestic Cat Given a taxonomy,we learn a metric for each internal (superclass) node n to discriminate between its subclasses. CanineFeline Wolf Dalmatian Big cat Approach the feature learning problem as hierarchical metric learning with disjoint regularization [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011

10. Further, we learn all metrics simultaneously with two regularizations A sparsity-based regularization to identify informative features. A disjoint regulazation to learn features exclusive to each granularity. Tree of Metrics Siamese cat Persian cat Carnivore Feline Domestic Cat Canine Wolf Dalmatian Big cat [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011

11. Regularization Terms to Learn Compact, Discriminative Metrics Minimize the sum of the diagonal entries. Sparsity regularization How can we select few informative features at each node? → Competition between features in a single metric [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011

12. Regularization Terms to Learn Compact, Discriminative Metrics How can we regularize each metric to use features disjoint from its ancestors? Disjoint regularization Enforce two metrics not to have large value at the same time, for the same feature. Ancestor Descendant → Competition between ancestors and descendants Both regularizers are convex [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011

13. Overview Goal: Learn and combine features across multiple taxonomies DalmatianWolf Siam. Cat Domestic leopard Wild Tameness DalmatianWolfLeopard Spotted Siam. Cat Pointy Corner Texture Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Dog-like shape Cat-face SpotPointy corner Indoor setting, person Woods Dog-like shape Cat-faceSpot Pointy corner Indoor setting, person Woods 1. Learn view and granularity specific features at each taxonomy 2. Optimally combine learned features for categorization +++ ++ Categorization model [Hwang12] S. J. Hwang, F. Sha, K. Grauman, Semantic Kernel Forest from Multiple Taxonomies, NIPS 2012

14. Semantic Kernel Forest DalmatianWolf Siam. Cat Domestic leopard Wild Tameness DalmatianWolfLeopard Spotted leopard Pointy Corner Texture Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Biological Appearance Habitat From multiple ToMs, we obtain a semantic kernel forest, a set of non-linear view- and granularity- specific feature spaces Compute RBF kernel on the distance computed using the learned metric [Hwang12] S. J. Hwang, F. Sha, K. Grauman, Semantic Kernel Forest from Multiple Taxonomies, NIPS 2012

15. Semantic Kernel Forest DalmatianWolf Siam. Cat Domestic leopard Wild Tameness DalmatianWolfLeopard Spotted leopard Pointy Corner Texture Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Biological Appearance Habitat How to combine the learned kernel forest for optimal discrimination? Obtain class specific kernel by linearly combining kernels on the tree paths. multiple kernel learning Consider only a small fraction of relevant kernels – O(TlogN)

16. Proposed Sparse Hierarchical Regularization Dalmatian Siam. Cat Wolfleopard Feline Animal Biological DalmatianWolf Siam. Cat Domestic leopard Wild Habitat Usual L1 regularization: selects few useful kernels Multiple taxonomies may provide some redundant kernels Canine Tameness - Interleaved selection of kernels Are all kernels equal?

17. Hierarchical regularization - weight of a node must be larger than its children’s Dalmatian Siam. Cat Wolf Canine leopard Feline Animal Biological < Proposed Sparse Hierarchical Regularization - Implicitly enforce hierarchical structure among kernels DalmatianWolf Siam. Cat Domestic leopard Wild Habitat < Tameness Multiple taxonomies provide redundant kernels - Higher level kernels discriminate with more categories

18. Optimization for Semantic Kernel Forest Nonsmooth due to the hierarchical regularization term MKL objective - Use projected subgradient to optimize We minimize the sum of the MKL objective + regularization term Sparsity Reg.Hierarchical regularization [Hwang12] S. J. Hwang, F. Sha, K. Grauman, Semantic Kernel Forest from Multiple Taxonomies, NIPS 2012

19. Datasets AWA-10 -6,180 images -10 animal classes -Fine-grained Imagenet-20 -28,957 images -20 non-animal classes -Coarser-grained (a) Wordnet(b) Appearance(c) Behavior(d) Habitat (a) Wordnet (b) Visual (c) Attributes Constructed on different attribute groups

20. Multiclass Classification Results MethodAWA-4AWA-10Imagenet-20 Raw feature kernel47.67 ± 2.2230.80 ± 1.3628.20 ± 1.45 Raw feature kernel + MKL48.50 ±1.89 31.13 ± 2.8127.67 ± 1.50 Perturbed semantic kernel treeN/A31.53 ± 2.0728.20 ± 2.02 We compare to three baselines - Raw feature kernel: RBF kernel computed on the original image features - Raw feature kernel + MKL: MKL with RBF kernels with different bandwidth. - Perturbed semantic kernel tree: Semantic kernel forest on randomly permuted taxonomy.

21. Multiclass Classification Results MethodAWA-4AWA-10Imagenet-20 Raw feature kernel47.67 ± 2.2230.80 ± 1.3628.20 ± 1.45 Raw feature kernel + MKL48.50 ±1.89 31.13 ± 2.8127.67 ± 1.50 Perturbed semantic kernel treeN/A31.53 ± 2.0728.20 ± 2.02 Semantic kernel tree + Avg47.17 ± 2.4031.92 ± 1.2128.97 ± 1.61 Semantic kernel tree + MKL48.89 ± 1.0632.43 ± 1.9329.74 ± 1.26 Semantic kernel tree + MKL-H50.06 ± 1.1232.68 ± 1.7929.90 ± 0.70 Semantic kernel tree (ToM) > perturbed kernel tree - Semantic kernel tree + Avg: Averged semantic kernel tree on a single taxonomy - Semantic kernel tree + MKL: MKL on a single taxonomy only with sparsity reg. - Semantic kernel tree + MKL-H: with both sparsity and hierarchical regularization. - More meaningful grouping/splits for object categorization

22. Multiclass Classification Results MethodAWA-4AWA-10Imagenet-20 Raw feature kernel47.67 ± 2.2230.80 ± 1.3628.20 ± 1.45 Raw feature kernel + MKL48.50 ±1.89 31.13 ± 2.8127.67 ± 1.50 Perturbed Semantic kernel treeN/A31.53 ± 2.0728.20 ± 2.02 Semantic kernel tree + Avg47.17 ± 2.4031.92 ± 1.2128.97 ± 1.61 Semantic kernel tree + MKL48.89 ± 1.0632.43 ± 1.9329.74 ± 1.26 Semantic kernel tree + MKL-H50.06 ± 1.1232.68 ± 1.7929.90 ± 0.70 Semantic kernel forest+MKL49.67 ± 1.1134.60 ± 1.7830.97 ± 1.14 Semantic kernel forest+MKL-H52.83 ± 1.6835.87 ± 1.2232.30 ± 1.00 - Semantic kernel forest+MKL: MKL with kernels learned on multiple taxonomies, with only the sparsity regularization - Semantic kernel forest+MKL-H: with both sparsity and hierarchical regularization. Multiple taxonomies > a single taxonomy - Each taxonomy provides complementary information

23. Multiclass Classification Results MethodAWA-4AWA-10Imagenet-20 Raw feature kernel47.67 ± 2.2230.80 ± 1.3628.20 ± 1.45 Raw feature kernel + MKL48.50 ±1.89 31.13 ± 2.8127.67 ± 1.50 Perturbed Semantic kernel treeN/A31.53 ± 2.0728.20 ± 2.02 Semantic kernel tree + Avg47.17 ± 2.4031.92 ± 1.2128.97 ± 1.61 Semantic kernel tree + MKL48.89 ± 1.0632.43 ± 1.9329.74 ± 1.26 Semantic kernel tree + MKL-H50.06 ± 1.1232.68 ± 1.7929.90 ± 0.70 Semantic kernel forest+MKL49.67 ± 1.1134.60 ± 1.7830.97 ± 1.14 Semantic kernel forest+MKL-H52.83 ± 1.6835.87 ± 1.2232.30 ± 1.00 Hierarchical regularizer > Standard L1 regularization - Regularizer’s effect is minimal on the semantic kernel tree, which lacks redundancy - Good to consider the structure of the feature spaces

24. Confusion matrices on 4 animal classes Blue: Low confusion Red: High confusion Each taxonomy is suboptimal, but provides complementary information which could be optimally leveraged with MKL CanineFeline Animal Biological DalmatianWolfSiam. CatLeopard Spotted Pointy Ear Animal Appearance Dalmatian Wolf Siam. Cat Leopard DomesticWild Animal DalmatianSiam. CatLeopardWolf Habitat

25. Lower Wordnet Higher Appearance BehaviorHabitat Effect of hierarchical regularization Sparsity regularization only: 34.33Sparsity+ Hierarchical: 35.67 Hierarchical regularizer avoids overfitting with implicit structure enforced among kernels procyonidfeline even-toed aquatic carnivore placental cat/rat hairless ~panda appearance racoon/rat land aquatic predator/prey behaviorjungle nonjungle aquatic land habitat

26. Key message: semantic taxonomies for visual feature learning Summary - Exploits disjoint sparsity between parent and child classes in a taxonomy: Tree of Metrics - Leverages complementary information from multiple semantic taxonomies: Semantic Kernel Forest - Novel regularizers that exploit category relations - Disjoint regularizer that exploits parent-child relationship to learn disjoint features. - Hierarchical regularizer that favors upper level kernels. [Hwang12] S. J. Hwang, F. Sha, K. Grauman, Semantic Kernel Forest from Multiple Taxonomies, NIPS 2012 [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011 Intuition Competing features between parent and child Complementary across different semantic views Learning Methods Disjoint and hierarchical regularizer for competing features MKL with hierarchical regularizer for complementary features

27. Key message: Semantic taxonomies for visual feature learning Summary [Hwang12] S. J. Hwang, F. Sha, K. Grauman, Semantic Kernel Forest from Multiple Taxonomies, NIPS 2012 [Hwang11] Learning a Tree of Metrics with Disjoint Visual Features, S. J. Hwang, F. Sha, and K. Grauman, NIPS 2011 Intuition: Competing features between parent and child Complementary across different semantic views Learning methods: Disjoint regularizer for competing features MKL with hierarchical regularizer for complementary features

28. A single taxonomy often improves performance on some classes, at the expense of others. - Individual taxonomy suboptimal. Habitat - Better for h. whale - Worse for panda Wordnet - Better for panda - Worse for h. whale All - Better for both Semantic kernel forest takes the best of both through learned combination. Per-class results

29. Idea Learn non-linear feature space for each view and granularity, that splits the categories according to each merge/split criteria Dog-like shape Cat-face Canine vs. Feline DalmatianWolf Siam. Cat Domestic leopard Wild Tameness DalmatianWolfLeopard Spotted Siam. Cat Pointy Corner Texture Appearance Habitat SpotPointy corner Indoor setting, person Woods

30. Idea Dog-like shape Cat-face Canine vs. Feline DalmatianWolf Siam. Cat Domestic leopard Wild Tameness Habitat Indoor setting, person Woods Spot vs. Pointy corner SpotPointy corner Learn non-linear feature space for each view and granularity, that splits the categories according to each merge/split criteria

31. Idea Dog-like shape Cat-face Canine vs. Feline Spot vs. Pointy corner SpotPointy corner Learn non-linear feature space for each view and granularity, that splits the categories according to each merge/split criteria Domestic vs. Wild Indoor setting, person Woods

32. Idea Dog-like shape Cat-face Spot vs. Pointy corner Domestic vs. Wild SpotPointy corner Indoor setting, person Woods Canine vs. Feline Then, combine the feature space to obtain an optimally discriminative space for categorization. Combined feature space Combined feature space How do we then, - learn such features, and - optimally combine them?