1. Data Mining: Principles and Algorithms — Chapter 10 — — Mining Multimedia Data—
©Jiawei Han
Department of Computer Science
University of Illinois at Urbana-Champaign
9/17/2018
Data Mining: Principles and Algorithms

2. Data Mining: Principles and Algorithms
9/17/2018
Data Mining: Principles and Algorithms

3. Mining Multi-Media Data
Similarity Search in Multimedia Data
Mining Multidimensional Multimedia Databases
Visual words: Pattern-based multimedia mining
Summary
9/17/2018
Data Mining: Principles and Algorithms

4. Similarity Search in Multimedia Data
Description-based retrieval systems
Build indices and perform object retrieval based on image descriptions, such as keywords, captions, size, and time of creation
Labor-intensive if performed manually
Results are typically of poor quality if automated
Content-based retrieval systems
Support retrieval based on the image content, such as color histogram, texture, shape, objects, and wavelet transforms
9/17/2018
Data Mining: Principles and Algorithms

5. Queries in Content-Based Retrieval Systems
Image sample-based queries
Find all of the images that are similar to the given image sample
Compare the feature vector (signature) extracted from the sample with the feature vectors of images that have already been extracted and indexed in the image database
Image feature specification queries
Specify or sketch image features like color, texture, or shape, which are translated into a feature vector
Match the feature vector with the feature vectors of the images in the database
9/17/2018
Data Mining: Principles and Algorithms

6. Approaches Based on Image Signature
Color histogram-based signature
The signature includes color histograms based on color composition of an image regardless of its scale or orientation
No information about shape, location, or texture
Two images with similar color composition may contain very different shapes or textures, and thus could be completely unrelated in semantics
Multifeature composed signature
Define different distance functions for color, shape, location, and texture, and subsequently combine them to derive the overall result
9/17/2018
Data Mining: Principles and Algorithms

7. Data Mining: Principles and Algorithms
Wavelet Analysis
Wavelet-based signature
Use the dominant wavelet coefficients of an image as its signature
Wavelets capture shape, texture, and location information in a single unified framework
Improved efficiency and reduced the need for providing multiple search primitives
May fail to identify images containing similar objects that are in different locations.
9/17/2018
Data Mining: Principles and Algorithms

8. One Signature for the Entire Image?
Walnus: [NRS99] by Natsev, Rastogi, and Shim
Similar images may contain similar regions, but a region in one image could be a translation or scaling of a matching region in the other
Wavelet-based signature with region-based granularity
Define regions by clustering signatures of windows of varying sizes within the image
Signature of a region is the centroid of the cluster
Similarity is defined in terms of the fraction of the area of the two images covered by matching pairs of regions from two images
9/17/2018
Data Mining: Principles and Algorithms

9. Mining Multi-Media Data
Similarity Search in Multimedia Data
Mining Multidimensional Multimedia Databases
Visual words: Pattern-based multimedia mining
Summary
9/17/2018
Data Mining: Principles and Algorithms

10. Multidimensional Analysis of Multimedia Data
Multimedia data cube
Design and construction similar to that of traditional data cubes from relational data
Contain additional dimensions and measures for multimedia information, such as color, texture, and shape
The database does not store images but their descriptors
Feature descriptor: a set of vectors for each visual characteristic
Color vector: contains the color histogram
MFC (Most Frequent Color) vector: five color centroids
MFO (Most Frequent Orientation) vector: five edge orientation centroids
Layout descriptor: contains a color layout vector and an edge layout vector
9/17/2018
Data Mining: Principles and Algorithms

11. Multi-Dimensional Search in Multimedia Databases
9/17/2018
Data Mining: Principles and Algorithms

12. Multi-Dimensional Analysis in Multimedia Databases
Color histogram
Texture layout
9/17/2018
Data Mining: Principles and Algorithms

13. Mining Multimedia Databases
Refining or combining searches
Search for “airplane in blue sky”
(top layout grid is blue and
keyword = “airplane”)
Search for “blue sky and
green meadows”
(top layout grid is blue
and bottom is green)
Search for “blue sky”
(top layout grid is blue)
9/17/2018
Data Mining: Principles and Algorithms

14. Mining Multimedia Databases
JPEG
GIF
Small
Very Large
RED
WHITE
BLUE
By Colour
By Format & Colour
By Format & Size
By Colour & Size
By Format
By Size
Sum
The Data Cube and
the Sub-Space Measurements
Medium
Large
Three Dimensions
Two Dimensions
RED
WHITE
BLUE
GIF
JPEG
By Format
By Colour
Sum
Cross Tab
Format of image
Duration
Colors
Textures
Keywords
Size
Width
Height
Internet domain of image
Internet domain of parent pages
Image popularity
Dimensions
RED
WHITE
BLUE
Colour
Sum
Group By
Measurement
9/17/2018
Data Mining: Principles and Algorithms

15. Mining Multimedia Databases in
MultiMediaMiner
9/17/2018
Data Mining: Principles and Algorithms

16. Classification in MultiMediaMiner
9/17/2018
Data Mining: Principles and Algorithms

17. Mining Associations in Multimedia Data
Special features:
Need # of occurrences besides Boolean existence, e.g.,
“Two red square and one blue circle” implies theme “air-show”
Need spatial relationships
Blue on top of white squared object is associated with brown bottom
Need multi-resolution and progressive refinement mining
It is expensive to explore detailed associations among objects at high resolution
It is crucial to ensure the completeness of search at multi-resolution space
9/17/2018
Data Mining: Principles and Algorithms

18. Mining Multimedia Databases
Spatial Relationships from Layout
property P1 on-top-of property P2
property P1 next-to property P2
Different Resolution Hierarchy
9/17/2018
Data Mining: Principles and Algorithms

19. Mining Multimedia Databases
From Coarse to Fine Resolution Mining
9/17/2018
Data Mining: Principles and Algorithms

20. Challenge: Curse of Dimensionality
Difficult to implement a data cube efficiently given a large number of dimensions, especially serious in the case of multimedia data cubes
Many of these attributes are set-oriented instead of single-valued
Restricting number of dimensions may lead to the modeling of an image at a rather rough, limited, and imprecise scale
More research is needed to strike a balance between efficiency and power of representation
9/17/2018
Data Mining: Principles and Algorithms

21. Mining Multi-Media Data
Similarity Search in Multimedia Data
Mining Multidimensional Multimedia Databases
Visual words: Pattern-based multimedia mining
Summary
9/17/2018
Data Mining: Principles and Algorithms

22. “Bag of Words”: Pattern-Mining + IVR
Object
Bag of ‘words’
View each object as a bag of words (similar to “IR”)
Bridge pattern mining with image & video recognition
Explore efficiency and diversity of pattern mining algorithms
9/17/2018
Data Mining: Principles and Algorithms

23. Data Mining: Principles and Algorithms
What Are “Bag of Words”?
Independent features
Histogram representation
9/17/2018
Data Mining: Principles and Algorithms

24. Data Mining: Principles and Algorithms
learning
recognition
feature detection
& representation
codewords dictionary
image representation
category
decision
category models
(and/or) classifiers
9/17/2018
Data Mining: Principles and Algorithms

25. Use “Bag of Words” in Recognition
9/17/2018
Data Mining: Principles and Algorithms

26. Exploring Spatial Information
Spatial info is often essential for quality recognition
Feature level
Discriminative methods
Generative models
Niebles & Fei-Fei, CVPR 2007 P3 P1 P2 P4 Bg
Image w
9/17/2018
Data Mining: Principles and Algorithms

27. Handling Invariance Issues
Scale and rotation
Implicit
Detectors and descriptors
Occlusion
Implicit in the models
Codeword distribution: small variations
Theme (z) distribution: different occlusion patterns
Translation
Encode (relative) location information
View point
Codewords: detector and descriptor
Theme distributions: different view points
9/17/2018
Data Mining: Principles and Algorithms

28. SpaRClus: Spatial Relationship Pattern-Based Hierarchical Clustering
A Simple “Bag of Words” Approach
Feature Detection
Feature Representation
Codebook Generation
Apply Document Clustering Algs
+ Text annotation, Color, Shape, Texture = Still not enough?!
Take care of rotation, scaling, translation?
Our solution: S. Kim, X. Jin and J. Han, “SpaRClus: Spatial Relationship Pattern-Based Hierarchical Clustering”, SDM'08
Violin and Guitar
9/17/2018
Data Mining: Principles and Algorithms 28

29. From Frequent Itemsets to Semantically Meaningful Visual Patterns
KDD 2007
Junsong Yuan, Ying Wu, Ming Yang
EECS, Northwestern Univ
9/17/2018
Data Mining: Principles and Algorithms 29

30. Persist Over Scaling, Translation, and Rotation?
How to perform Image Clustering that persists over Scaling, Translation, and Rotation transformations?
Make good use of
Bag of Items
Spatial Information
But HOW???
(a) original pattern
(b) rotated pattern
Need to study
What is not changed over such transformations!
(c) scaled pattern
(d) translated pattern
9/17/2018
Data Mining: Principles and Algorithms 30

31. Data Mining: Principles and Algorithms
Spatial Pattern
Define a 3-pattern p as
p = (<a1,a2,a3>,θ, r) where
r = d(c,a3)/d(a1,a2)
Define a (spatial) pattern as
a set of 3-patterns a3 a2 a1 c θ A B A B A B C C D D
9/17/2018
Data Mining: Principles and Algorithms 31

32. Data Mining: Principles and Algorithms
3-pattern
Basic unit of a spatial pattern
Need an approximation to group 3-patterns
Group similar 3-patterns
Need to have same item bags
θ and r should be within given thresholds a3 a2 a1 c θ c’ θ’
9/17/2018
Data Mining: Principles and Algorithms 32

33. Data Mining: Principles and Algorithms
SpIBag
SpIBag (Spatial Item Bag Mining)
Find frequent spatial patterns
Each image is made up of 3-patterns
Apply the Apriori algorithm (or any other good frequent pattern mining algorithm)
Special considerations on Joining step a2 a1 a3
(a) Two joinable 3-patterns
(b) Two non-joinable 3-patterns a4
9/17/2018
Data Mining: Principles and Algorithms 33

34. Data Mining: Principles and Algorithms
SpaRClus
SpaRClus (Spatial Relationship Pattern-Based Hierarchical Clustering)
Apply SpiBag + Pruning
Merge leaves of the graph
Use the same Entropy function to see the tightness of two clusters
Make clusters disjoint
Use the score function of an image to a cluster
9/17/2018
Data Mining: Principles and Algorithms 34

35. Experiments (Kitchen Plan Images)
9/17/2018
Data Mining: Principles and Algorithms 35

36. Exploration of the Power of Pattern Mining in Image and Video Analysis
SpaRClus: Find invariant that persists over Scaling, Translation, and Rotation transformations
“Bag of word”: simplest kind of patterns in pattern mining
What about “subsequences”, “substructures”, and many other patterns?
Scalable algorithms integrated with powerful pattern recognition methods
Beyond clustering, classification, correlation analysis tasks?
9/17/2018
Data Mining: Principles and Algorithms 36

37. Mining Multi-Media Data
Similarity Search in Multimedia Data
Mining Multidimensional Multimedia Databases
Visual words: Pattern-based multimedia mining
Summary
9/17/2018
Data Mining: Principles and Algorithms

38. Data Mining: Principles and Algorithms
Summary
Mining object data needs feature/attribute-based generalization methods
Spatial, spatiotemporal and multimedia data mining is one of important research frontiers in data mining with broad applications
Spatial data warehousing, OLAP and mining facilitates multidimensional spatial analysis and finding spatial associations, classifications and trends
Multimedia data mining needs content-based retrieval and similarity search integrated with mining methods
9/17/2018
Data Mining: Principles and Algorithms

39. Some References on Multimedia Data Mining
Alan C. Bovik, Handbook of Image and Video Processing, Academic Press, 2005
Francesco Camastra and Alessandro Vinciarelli, Machine Learning for Audio, Image and Video Analysis: Theory and Applications, Springer 2008
Ritendra Datta, Dhiraj Joshi, Jia Li, James Z. Wang (2008). Image Retrieval: Ideas, Influences, and Trends of the New Age. ACM Computing Surveys. 
Sangkyum Kim, Xin Jin and Jiawei Han. SpaRClus: Spatial Relationship Pattern-Based Hierarchical Clustering. SDM'08
Michael Lew, et al., Content-based Multimedia Information Retrieval: State of the Art and Challenges, ACM Transactions on Multimedia Computing, Communications, and Applications, pp. 1-19, 2006.
C. Niebles, H. Wang and L. Fei-Fei. Unsupervised learning of human action categories using spatial-temporal words. International Journal of Computer Vision. 79(3)
Valery A. Petrushin and Latifur Khan (eds)., Multimedia Data Mining and Knowledge Discovery, Springer 2006
Azriel Rosenfeld, David Doermann, Daniel DeMenthon (eds)., Video Mining, Springer 2003
Junsong Yuan, Ying Wu, and Ming Yang. From frequent itemsets to semantically meaningful visual patterns. KDD '07
9/17/2018
Data Mining: Principles and Algorithms

40. Data Mining: Principles and Algorithms
9/17/2018
Data Mining: Principles and Algorithms