1. Image Classification for Automatic Annotation
Jianping Fan
Department of Computer Science
University of North Carolina at Charlotte

2. Feature Extraction Object-Based Approach Visual Features Input Image
Salient Objects
Color histogram, Tamura texture,
Locations …….
Visual Features

3. Feature Extraction
Image-Based Approach

4. Image Representation Images Salient Objects Color Color histogram
Tamura Texture
Wavelet
Texture
histogram
Shape
Interest
Point
set

5. Feature-Based Image Representation
dimension
Feature
dimension
Feature
dimension
Curse of Dimensionality

6. Feature-Based Image Representation
Tree-Based Database Indexing

7. Feature-Based Image Representation
Tree-Based Database Indexing
Nearest Neighbor Search
Overlapping between
Different nodes
What’s the solution?

8. Feature-Based Image Representation
Where idea for tree-based indexing come from?
Library:
12,000,000 books

9. Feature-Based Image Representation
Books in Library
Natural Sciences
Social Sciences
Computer
Science
Electrical
Engineering
Dancing
Computer
Languages
Researches
I get it!
Database
Multimedia
Too easy!
11!

10. Feature-Based Image Representation
What is the solution?
Let's go back to pinciple!
Concept Hierarchy or Ontology

11. Hierarchical Concept Organization
Concept Ontology

12. Hierarchical Concept Organization
Concept Ontology

13. 6. Classifier Training for Atomic Image Concepts
Beach
Atomic Image Concept
Different Patterns
of Co-Appearances
of Salient Objects
Water & Sky
Water & Sand
Water, Sand, Sky, …
Feature
Subset 9
Feature
Subset 1
Feature
Subset 2
Feature Subsets

14. 6. Classifier Training for Atomic Image Concepts
Curse of Dimensionality
# samples needed increase with # dimensions (generally exponentially) .
Human labeling is expensive
Some features are redundant
Proposal
Joint SVM boosting and feature selection

15. Subspace 1 Subspace 2 Subspace 3 … Subspace N Higher Level Classifier
Boosting SVM Classifier Training
PCA
PCA
PCA
PCA
High dimensional feature space
Subspace Subspace Subspace 3 … Subspace N
SVM
SVM
SVM
SVM
Low-level classifiers …
Weak Classifier 1
Weak Classifier 2
Weak Classifier 3
Weak Classifier N
Boosting for optimal combination
Higher Level Classifier
High-level classifier
Less training samples due to dimension reduction
Reuse training results on low-level concepts
More selection opportunities compared to filter and wrapper

16. 6. Classifier Training for Atomic Image Concepts
Kernel-Based Data Warping
Kernel Function:

17. 6. Classifier Training for Atomic Image Concepts
Kernel for Color Histogram
Statistical Image Similarity
Kernel

18. 6. Classifier Training for Atomic Image Concepts
Wavelet Filter Bank Kernel

19. 6. Classifier Training for Atomic Image Concepts
Wavelet Filter Bank Kernel

20. 6. Classifier Training for Atomic Image Concepts
Interest Point Matching Kernel

21. 6. Classifier Training for Atomic Image Concepts
Interest Point Matching Kernel

22. 6. Classifier Training for Atomic Image Concepts
Multiple Kernel Learning
SVM Image Classifier

23. 6. Classifier Training for Atomic Image Concepts
Dual Problem
Subject to:

24. 6. Classifier Training for Atomic Image Concepts

25. 6. Classifier Training for Atomic Image Concepts
Some Results
Beach Scene
Garden Scene

26. High-Level Image Concept Modeling
Inter-Concept Similarity Modeling
Nature Scene
Flower View
Garden
Beach
Nature Scene: Larger Hypothesis Space & Large Variations of Visual Properties!
Garden, Beach, Flower view: Different but share common visual properties!

27. 7. Classifier Training for High-Level Image Concepts
Challenging Problems
Error Transmission Problems
Training Cost Issue
Knowledge Transferability and Task Relatedness Exploitation

28. 7. Classifier Training for High-Level Image Concepts
Error Transmission Problem
The classifiers for low-level image concepts cannot recover the errors for
the classifiers of high-level image concepts!

29. 7. Classifier Training for High-Level Image Concepts
Error Transmission Problem
Outdoor
Flower View
Garden
Beach
Errors for the classifiers of atomic image concepts may be transmitted to
the classifiers for the high-level image concepts!

30. 7. Classifier Training for High-Level Image Concepts
Training Cost Issue
Multiple Hypotheses
outdoor
garden
flower view
beach
Large Diversity of Contents

31. 7. Classifier Training for High-Level Image Concepts
Knowledge Transferability & Task Relatedness Exploitation
Outdoor
Flower View
Garden
Beach
They are different but strongly related!

32. 7. Classifier Training for High-Level Image Concepts
Multi-Task Learning
Which tasks are strongly related?
How to quantify the task relatedness?
How to integrate such task relatedness for training large-scale related image classifiers?

33. 7. Classifier Training for High-Level Image Concepts
Related Learning Tasks
Nature Scene
Flower View
Garden
Beach
They are different but strongly related!
Concept Ontology can provide a good
environment for multi-task learning!

34. 7. Classifier Training for High-Level Image Concepts
Related Learning Tasks

35. 7. Classifier Training for High-Level Image Concepts
Relatedness Modelling
outdoor
garden
flower view
beach
: Common Prediction Structure

36. 7. Classifier Training for High-Level Image Concepts
Joint Objective Function
Subject to:

37. 7. Classifier Training for High-Level Image Concepts
Dual Problem
Subject to:

38. 7. Classifier Training for High-Level Image Concepts
Biased Classifier Training
Dual Problem
Subject to:

39. 7. Classifier Training for High-Level Image Concepts
Common Prediction Structure
Nature Scene
Flower View
Garden
Beach
Common Visual Properties

40. 7. Classifier Training for High-Level Image Concepts
Hierarchical Boosting

41. 7. Classifier Training for High-Level Image Concepts
Biased Classifier for Parent Node

42. 7. Classifier Training for High-Level Image Concepts
Hierarchical Boosting to Generate Classifier for Parent Node

43. 7. Classifier Training for High-Level Image Concepts
Performance Evaluation

44. 7. Classifier Training for High-Level Image Concepts
Performance Evaluation

45. 7. Classifier Training for High-Level Image Concepts
Advantages of Hierarchical Boosting
Handling inter-concept similarity via multi-task learning
Reducing training cost
Enhancing discrimination power of the classifiers

46. 8. Hierarchical Image Classification
Overall Probability
Parent Node
Path 1
Path 2
Path C
Children Node 1
Children Node 2
Children Node C

47. 8. Hierarchical Image Classification
Some Results

48. 10. Query Result Evaluation
Allow Users to See Global View!

49. 10. Query Result Evaluation
Allow Users to See Similarity Direction!

50. 10. Query Result Evaluation
Allow Users to Zoom into Images of Interest!

51. 10. Query Result Evaluation: Red Flower
Allow Users to Select Query Example Interactively!

52. 10. Query Result Evaluation: Sunset
Allow Users to Look for Particular Images!

53. 11. Training Image Observation

54. 11. Training Image Observation

55. 11. Training Image Observation