1. Image Segmentation Techniques
Advanced Multimedia Databases
Jeff Strickrott

2. Content Introduction Image segmentation methods Summary Template
Transformation based
Probabilistic modeling
Summary
11/19/2018
Image Segmentation

3. Introduction What is the goal behind building multimedia databases?
We want to ask questions based on content.
Of the form: what images show the red car in front of the house?
Why do we need to segment image?
To represent content! Content Represented by objects or spatial regions in the image.
To support region/content based queries.
11/19/2018
Image Segmentation

4. Template Based Segmentation
A template is a generalization of the features of some object.
Spatially distributed.
Features can be pixel: intensity, color, relations between regions, etc.
Detect object through correlation (match object to template).
Detect objects the same size as template.
Requires domain knowledge (to create template).
Result: Provides semantic segmentation of image.
11/19/2018
Image Segmentation

5. Face Templates Examples
Scasselati Ratio-Metric
Template [Scaz 98]
Liu’s Intensity Based Template [Liu 00]
Two templates for real-time face detection: Average intensity &
Ratio-metric
11/19/2018
Image Segmentation

6. Using Templates Algorithm (Naive):
Select region of image.
Slide template over region, moving “i” pixels at a time.
Compute correspondence for each location.
Locations with high correspondence are locations of objects.
Resize (Downsample) to find faces of different sizes.
Repeat
Dynamic programming can be used to make search more efficient (Liu 00).
11/19/2018
Image Segmentation

7. Results Liu’s & Wang’s Face Temp.
Video Scene Segmentation: on new face start new scene.
11/19/2018
Image Segmentation

8. Transformation Based Segmentation: Image Mapping
Map Image from intensity space to new feature space.
Reason: Easier to detect relevant information about objects in the image.
Transform Examples: Fourier Transform, DCT, Wavelet.
11/19/2018
Image Segmentation

9. Wavelet Image Representation
Maps 2D image into varying spatial and frequency resolution space.
Haar Basis function encodes relationship between neighboring pixel intensities (edges).
O(n), n = the number of pixels.
Linear transform
(invertible/lossless).
11/19/2018
Image Segmentation

10. Poggio’s People Detection System
Generate Templates of Objects in Domain from over-complete Haar wavelet.
Averaged over transformed images of objects to get relevant features (multiple views).
Spatial correspondence between transform coefficients and object location.
Template encodes relationship between regions in image (absolute or ratio of intensities).
Encode spatial relationship.
11/19/2018
Image Segmentation

11. Standard Haar Wavelet L = Lowpass H = Highpass XYz X Row Y Column
Z level
11/19/2018
Image Segmentation

12. People Templates Each square one wavelet Black: Relevant edge
Gray: random patterns
Black: Relevant edge
information
[Poggio 00]
11/19/2018
Image Segmentation

13. Detection System Algorithm: Proven also on faces and cars.
Image transformed.
Template matching done in wavelet space.
Shift of wavelet coefficient = 4 pixels in image.
Image must be resized to find objects not the size of template.
Proven also on faces and cars.
Real time people detection in
complex scene.
[Poggio 00]
11/19/2018
Image Segmentation

14. Probabilistic Modeling
Generic Model, No Segmentation. We know nothing about image domain.
Need generic criteria that captures local and global information.
Should perform well on texture and high color images.
Should work on sections of images.
No assumptions about similarity metric or dimension of feature space.
11/19/2018
Image Segmentation

15. Retrieval as Classification Problem
Model content of images as class(es) in some set of classes.
Bayesian classifier used to minimize the misclassification of image content.
Solve problem:
g*(X) = argmaxi P(X | Y=i) P(Y =i).
X can be an image, subset of image, other types of data (text, audio, etc.).
Will work on any type of features used to represent a class.
11/19/2018
Image Segmentation

16. Embedded Mixture Model
Vasconcelos showed feature representation choice is major factor in minimizing classification error.
Proposed solution to Bayesian min problem:
DCT image in blocks (separate or overlapping)
Model as Mixture of Gaussian distributions, this is a weighted sum of Gaussian probability densities.
11/19/2018
Image Segmentation

17. EMM Performance
Can model content from images with complex color and texture features (outperforms histogram based and texture based models for retrieval).
Works on compressed images.
Supports region based queries.
Classification algorithm runs in O(C2) time. C on the order of 8-16.
Prior probability term P(Y =i) can be used for learning via relevance feedback.
Expectation Maximization algorithm used for learning weight, mean and covariance parameters (w, m,S). More work must be done to make this efficient.
11/19/2018
Image Segmentation

18. Region Based Retrieval
Content based
Retrieval on a
Region of an
image.
Database of
mixed objects
(in color).
Results of
query on Onion
[Vasc 00]
11/19/2018
Image Segmentation

19. Summary
We have investigated three techniques for segmenting images that require varying amounts of domain knowledge.
Closest similarity with segmentation methods discussed in class are the clustering and stochastic modeling methods.
Two methods work in real-time while fourth designed for generic domains and offline classification.
11/19/2018
Image Segmentation

20. References
[Liu 00] Liu, Z. and Wang, Y., "Face Detection and Tracking in Video Using Dynamic Programming," ICIP-2000, Vancouver, Canada.
[Poggio 00] Papageorgiou, C. and Poggio, T., A Trainable System for Object Detection International journal of computer vision, vol. 38, pp , 2000.
[Scaz 98] Scassellati, B., "Eye Finding via Face Detection for a Foveated, Active Vision System," AAAI 98
[Stau 99] Stauffer, C. and Grimson, W. E. L., "Adaptive Background Mixture Models for Real-Time Tracking," IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246, 1999.
[Vasc 00]Vasconcelos, N. and Lippman, A., "A Probabilistic Architecture for Content-based Image Retrieval," IEEE Conference on Computer Vision and Pattern Recognition (ICPR 2000), Hilton Head Island, SC, 2000.
11/19/2018
Image Segmentation

21. Segmentation Methods Color Histogram Split and Merge Region Growing
segment in histogram space
no domain knowledge
Split and Merge
Segment image into regions ()
Region Growing
Grow regions around seed point based on similarity measure.
Similarity criteria
Clustering
Stochastic
11/19/2018
Image Segmentation