1. Image Classification over Visual Tree Jianping Fan Dept of Computer Science UNC-Charlotte, NC 28223 www.cs.uncc.edu/~jfan

2. 1. Research Motivation Large-Scale Visual Recognition Inter-concept Visual Correlations rather than independency

3. 1. Research Motivation Large-Scale Visual Recognition: Challenges We need to learn large amounts of classifiers for large-scale visual recognition! Some object classes and image concepts are visually-related and hard to be discriminated! Some object classes and image concepts may have huge inner-concept visual diversity!

4. 1. Research Motivation Large-Scale Visual Recognition: Challenges Huge inner-concept visual diversity ---simple models may not work, but using complex models may overlap with others! Huge inter-concept visual similarity ---training complexity will increase for distinguishing visually-related concepts! Huge computational cost ---thousands of inter-related classifiers should be trained jointly!

5. 2. Collecting Large-Scale Training Images Flickr Images & Other Image Sites Keywords for image crawling

6. Visual Feature Extraction 2. Collecting Large-Scale Training Images

7. Synonymous Concepts: Visual Similarity CVPR2010

8. Synonymous Concepts: Visual Similarity

9. Ambiguous Concept: Visual Diversity 2. Collecting Large-Scale Training Images

10. Ambiguous Concept: Visual Diversity 2. Collecting Large-Scale Training Images

11. Junk Image Filtering subject to: Decision function: R Outliers Majority IEEE Trans. CSVT 2009 2. Collecting Large-Scale Training Images

12. Junk Image Filtering 2. Collecting Large-Scale Training Images

13. Junk Image Filtering 2. Collecting Large-Scale Training Images

14. Most text terms are weakly related or even irrelevant to web images in the same webpage tiger big cat Southeast Asia Russian Chinese Bengal Siberian Indochinese South Chinese …. Noise image

15. Text-Image Alignment for Web Image Indexing WWW2010, PR2014 2. Large-Scale Image Preparation

16. Informative Image Extraction Noise image n Two simple rules: n Aspect ratio (>0.2 or <5) n Image size (min(width, height) > 60 pixel) n Not perfect but produce satisfied results n Unsupervised and computationally efficient

17. Webpage Segmentation Surrounding Text Extraction n Visual-based algorithm n precise but expensive n [Cai et al. MSR-TR’03] n DOM (Document Object Model) based method n computationally efficient

18. Webpage Segmentation Surrounding Text Extraction n Visual-based algorithm n precise but expensive n [Cai et al. MSR-TR’03] n DOM (Document Object Model) based method n computationally efficient

19. Text-Image Alignment for Web Image Indexing WWW2010, PR2014 Near-duplicates share similar semantics!

20. Text-Image Alignment for Web Image Indexing WWW2010, PR2014

21. Duplicate Detection CVPR2012 2. Collecting Large-Scale Training Images Duplicates may mislead classifier training tools!

22. Duplicate Detection 2. Collecting Large-Scale Training Images

23. Automatic Tag-Instance Alignment Missing Tag Prediction ACM MM 2010 CVPR 2012 2. Collecting Large-Scale Training Images

24. 3. Visual Concept Network Why we need visual concept network? ---concept ontology, object co-occurrence network, …. Common space: classifier training & concept detection ---visual feature space rather than label space or concept space We need to characterize inter-concept visual correlations rather than others! ACM MM2009 Inter-related learning task determination

25. 3. Visual Concept Network

30. Label Tree for Efficient Classification 1, 2, 3, 4 1 1, 3 2, 4 324 Label 1: cat Label 2: mini van Label 3: dog Label 4: fire truck It is a fire truck! Number of dot products needed in the label tree: 1 + 1 = 2 Number of dot product needed in a flat approach: 1 + 1 + 1 + 1 = 4 [Bengio et al. NIPS’2010]

31. Construction of Label Tree SVM 1 Testing Samples SVM 2 SVM N … Confusion Matrix 1, 2, …,10 1,5, 6 2, 4,8, 9 3,7, 10 2, 8 4,9 4 9 Training N one-vs-rest SVMs is very expensive n The SVMs could be unreliable n Huge sample imbalance n Negative samples could mislead the classifier training

32. Visual Similarity Matrix Result is based on ImageNet data set of 1000 categories

33. 4. Visual Tree Construction: Hierarchical Clustering Root

34. 4. Visual Tree Construction: Hierarchical Clustering

39. Bag-of-Words (BoW)

40. To distinguish visually-similar categories, dictionaries with strong discrimination is critical Joint dictionary learning CVPR 2012 TPAMI2013 5. Joint Dictionary Learning for Discriminative Image Representation

41. 7. Large-Scale Classifier Training IEEE Trans. IP 2011, IEEE Trans. PAMI 2014, PR 2013

42. Inference Model Selection for Classifier Training 7. Large-Scale Classifier Training

43. Inference Model Selection for Classifier Training 7. Large-Scale Classifier Training

45. Hierarchical Organization

46. 7. Large-Scale Classifier Training Hierarchical Organization

47. 7. Large-Scale Classifier Training Hierarchical Organization

48. 7. Large-Scale Classifier Training Flat Organization

49. 8. Interactive Classifier Assessment VAST 2011

50. 8. Interactive Classifier Assessment

52. 9. Some Experimental Results

60. 10. Conclusions & Current Work A new image representation approach via discriminative dictionary learning; A novel approach for semantic gap quantification; A new solution for inter-related classifier training and large-scale visual recognition. Performing our experiments on large-scale images !