Image Classification for Automatic Annotation Jianping Fan Department of Computer Science University of North Carolina at Charlotte http://www.cs.uncc.edu/~jfan

Feature Extraction Object-Based Approach Visual Features Input Image Salient Objects Color histogram, Tamura texture, Locations ……. Visual Features

Feature Extraction Image-Based Approach

Image Representation Images Salient Objects Color Color histogram Tamura Texture Wavelet Texture histogram Shape Interest Point set

Feature-Based Image Representation dimension Feature dimension Feature dimension Curse of Dimensionality

Feature-Based Image Representation Tree-Based Database Indexing

Feature-Based Image Representation Tree-Based Database Indexing Nearest Neighbor Search Overlapping between Different nodes What’s the solution?

Feature-Based Image Representation Where idea for tree-based indexing come from? Library: 12,000,000 books

Feature-Based Image Representation Books in Library Natural Sciences Social Sciences Computer Science Electrical Engineering Dancing Computer Languages Researches I get it! Database Multimedia Too easy! 11!

Feature-Based Image Representation What is the solution? Let's go back to pinciple! Concept Hierarchy or Ontology

Hierarchical Concept Organization Concept Ontology

Hierarchical Concept Organization Concept Ontology

6. Classifier Training for Atomic Image Concepts Beach Atomic Image Concept Different Patterns of Co-Appearances of Salient Objects Water & Sky Water & Sand Water, Sand, Sky, … Feature Subset 9 Feature Subset 1 Feature Subset 2 Feature Subsets

6. Classifier Training for Atomic Image Concepts Curse of Dimensionality # samples needed increase with # dimensions (generally exponentially) . Human labeling is expensive Some features are redundant Proposal Joint SVM boosting and feature selection

Subspace 1 Subspace 2 Subspace 3 … Subspace N Higher Level Classifier Boosting SVM Classifier Training PCA PCA PCA PCA High dimensional feature space Subspace 1 Subspace 2 Subspace 3 … Subspace N SVM SVM SVM SVM Low-level classifiers … Weak Classifier 1 Weak Classifier 2 Weak Classifier 3 Weak Classifier N Boosting for optimal combination Higher Level Classifier High-level classifier Less training samples due to dimension reduction Reuse training results on low-level concepts More selection opportunities compared to filter and wrapper

6. Classifier Training for Atomic Image Concepts Kernel-Based Data Warping Kernel Function:

6. Classifier Training for Atomic Image Concepts Kernel for Color Histogram Statistical Image Similarity Kernel

6. Classifier Training for Atomic Image Concepts Wavelet Filter Bank Kernel

6. Classifier Training for Atomic Image Concepts Wavelet Filter Bank Kernel

6. Classifier Training for Atomic Image Concepts Interest Point Matching Kernel

6. Classifier Training for Atomic Image Concepts Interest Point Matching Kernel

6. Classifier Training for Atomic Image Concepts Multiple Kernel Learning SVM Image Classifier

6. Classifier Training for Atomic Image Concepts Dual Problem Subject to:

6. Classifier Training for Atomic Image Concepts

6. Classifier Training for Atomic Image Concepts Some Results Beach Scene Garden Scene

High-Level Image Concept Modeling Inter-Concept Similarity Modeling Nature Scene Flower View Garden Beach Nature Scene: Larger Hypothesis Space & Large Variations of Visual Properties! Garden, Beach, Flower view: Different but share common visual properties!

7. Classifier Training for High-Level Image Concepts Challenging Problems Error Transmission Problems Training Cost Issue Knowledge Transferability and Task Relatedness Exploitation

7. Classifier Training for High-Level Image Concepts Error Transmission Problem The classifiers for low-level image concepts cannot recover the errors for the classifiers of high-level image concepts!

7. Classifier Training for High-Level Image Concepts Error Transmission Problem Outdoor Flower View Garden Beach Errors for the classifiers of atomic image concepts may be transmitted to the classifiers for the high-level image concepts!

7. Classifier Training for High-Level Image Concepts Training Cost Issue Multiple Hypotheses outdoor garden flower view beach Large Diversity of Contents

7. Classifier Training for High-Level Image Concepts Knowledge Transferability & Task Relatedness Exploitation Outdoor Flower View Garden Beach They are different but strongly related!

7. Classifier Training for High-Level Image Concepts Multi-Task Learning Which tasks are strongly related? How to quantify the task relatedness? How to integrate such task relatedness for training large-scale related image classifiers?

7. Classifier Training for High-Level Image Concepts Related Learning Tasks Nature Scene Flower View Garden Beach They are different but strongly related! Concept Ontology can provide a good environment for multi-task learning!

7. Classifier Training for High-Level Image Concepts Related Learning Tasks

7. Classifier Training for High-Level Image Concepts Relatedness Modelling outdoor garden flower view beach : Common Prediction Structure

7. Classifier Training for High-Level Image Concepts Joint Objective Function Subject to:

7. Classifier Training for High-Level Image Concepts Dual Problem Subject to:

7. Classifier Training for High-Level Image Concepts Biased Classifier Training Dual Problem Subject to:

7. Classifier Training for High-Level Image Concepts Common Prediction Structure Nature Scene Flower View Garden Beach Common Visual Properties

7. Classifier Training for High-Level Image Concepts Hierarchical Boosting

7. Classifier Training for High-Level Image Concepts Biased Classifier for Parent Node

7. Classifier Training for High-Level Image Concepts Hierarchical Boosting to Generate Classifier for Parent Node

7. Classifier Training for High-Level Image Concepts Performance Evaluation

7. Classifier Training for High-Level Image Concepts Performance Evaluation

7. Classifier Training for High-Level Image Concepts Advantages of Hierarchical Boosting Handling inter-concept similarity via multi-task learning Reducing training cost Enhancing discrimination power of the classifiers

8. Hierarchical Image Classification Overall Probability Parent Node Path 1 Path 2 Path C Children Node 1 Children Node 2 Children Node C

8. Hierarchical Image Classification Some Results

10. Query Result Evaluation Allow Users to See Global View!

10. Query Result Evaluation Allow Users to See Similarity Direction!

10. Query Result Evaluation Allow Users to Zoom into Images of Interest!

10. Query Result Evaluation: Red Flower Allow Users to Select Query Example Interactively!

10. Query Result Evaluation: Sunset Allow Users to Look for Particular Images!

11. Training Image Observation

11. Training Image Observation

11. Training Image Observation