1. SWE 423: Multimedia Systems
Chapter 4: Graphics and Images (3)

2. Outline Image Processing Basics Textbook Section 4.3
Image Features
Image Segmentation
Textbook Section 4.3
Additional Reference: Wasfi Al-Khatib, Y. Francis Day, Arif Ghafoor, and P. Bruce Berra. Semantic modeling and knowledge representation in multimedia databases. IEEE Transactions on Knowledge and Data Engineering, 11(1):64-80, 1999.

3. Image Processing
Image processing involves the analysis of scenes or the reconstruction of models from images representing 2D or 3D objects.
Image Analysis
Identifying Image Properties (Image Features)
Image Segmentation
Image Recognition
We will look at image processing from a database perspective.
Objective: Design of robust image processing and recognition techniques to support semantic modeling, knowledge representation, and querying of images.

4. Semantic Modeling and Knowledge Representation in Image Databases
Feature Extraction.
Salient Object Identification.
Content-Based Indexing and Retrieval.
Query Formulation and Processing.

5. Multi-Level Abstraction
Semantic Modeling
And Knowledge
Representation
Layer
Semantic Specification
Knowledge Base
Semantic Identification
Process
Object
Recognition
Layer
Object Models
Object Recognition
Process
Feature
Extraction
Layer
Feature Specification
Feature Extraction
Process
Image Data
Still Video Frames
Multimedia
Data

6. Feature Extraction Layer
Image features: Colors, Textures, Shapes, Edges, ...etc.
Features are mapped into a multi-dimensional feature space allowing similarity-based retrieval.
Features can be classified into two types: Global and Local.

7. Global Features
Generally emphasize coarse-grained pattern matching techniques.
Transform the whole image into a functional representation.
Finer details within individual parts of the image are ignored.
Examples: Color histograms and coherence vectors, Texture, Fast Fourier Transform, Hough Transform, and Eigenvalues.
What are some of the example queries?

8. Color Histogram How many pixels of the image take a specific color
In order to control the number of colors, the domain is discretized
E.g. consider the value of the two leftmost bits in each color channel (RGB).
In this case , the number of different colors is equal to __________
How can we determine whether two images are similar using the color histogram?

9. Color Coherence Vector
Based on the color histogram
Each pixel is checked as to whether it is within a sufficiently large one-color environment or not.
i.e. in a region related by a path of pixels of the same color
If so, the pixel is called coherent, otherwise incoherent
For each color j, compute the number of coherent and incoherent pixels (j , j), j = 1, ..., J
When comparing two images with color coherence vectors (j , j) and (j , j), j = 1, ..., J, we may use the expression

10. Texture Texture is a small surface structure Examples include
Natural or artificial
Regular or irregular
Examples include
Wood barks
Knitting patterns
The surface of a sponge

11. Texture Examples Artificial/periodic Artificial/non-periodic
Photographic/pseudo-periodic
Photographic/random
Photographic/structured
Inhomogeneous (non-texture)

12. Texture Two basic approaches to study texture
Structural analysis searches for small basic components and an arrangement rule
Statistical analysis describes the texture as a whole based on specific attributes (local gray-level variance, regularity, coarseness, orientation, and contrast.
Either done in the spatial domain or the spatial frequency domain

13. Global Features Advantages: Disadvantages: Simple.
Low computational complexity.
Disadvantages:
Low accuracy

14. Local Features
Images are segmented into a collection of smaller regions, with each region representing a potential object of interest (fine-grained).
An object of interest may represent a simple semantic object (e.g. a round object).
Choice of features is domain specific:
X-ray imaging, GIS, ...etc require spatial features (e.g. shapes [may be calculated through edges] and dimensions.)
Paintings, MMR imaging, ...etc may use color features in specific regions of the image.

15. Edge Detection
A given input image E is used to gradually compute a (zero-initialized) output image A.
A convolution mask runs across E pixel by pixel and links the entries in the mask at each position that M occupies in E with the gray value of the underlying image dots.
The result of the linkage (and the subsequent sum across all products from the mask entry and the gray value of the underlying image pixel) is written to the output image A.

16. Convolution
Convolution is a simple mathematical operation which is fundamental to many common image processing operators.
Convolution provides a way of `multiplying together' two arrays of numbers, generally of different sizes, but of the same dimensionality, to produce a third array of numbers of the same dimensionality.
This can be used in image processing to implement operators whose output pixel values are simple linear combinations of certain input pixel values.
The convolution is performed by sliding the kernel over the image, generally starting at the top left corner, so as to move the kernel through all the positions where the kernel fits entirely within the boundaries of the image.

17. Convolution Computation
If the image E has M rows and N columns, and the kernel K has m rows and n columns, then the size of the output image A will have M - m + 1 rows, and N - n + 1 columns and is given by:
Example page 60.

18. Similarity Metrics Minkowski Distance Weighted Distance
Average Distance
Color Histogram Intersection

19. Prototype Systems QBIC (http://www.hermitagemuseum.org)
Uses color, shape, and texture features
Allows queries by sketching features and providing color information
Chabot (Cypress)
Uses color and textual annotation.
Improved performance due to textual annotation (Concept Query)
KMeD
Uses shapes and contours as features.
Features are extracted automatically in some cases and manually in other cases.

20. Demo (Andrew Berman & Linda G. Shapiro )