The MPEG-7 Color Descriptors By Bassam kehail 120060647

Outline CLD extraction Color Structure Descriptor (CSD) CSD extraction Introduction Visual Descriptors (color, texture, shape) Scalable Color Descriptor (CSD) SCD extraction GoF/GoP Color Descriptor Dominant Color Descriptor (DCD) DCD Extraction Color Layout Descriptor (CLD) CLD extraction Color Structure Descriptor (CSD) CSD extraction CSD scaling Summary

Introduction Color is an important visual attribute for both human vision and computer processing. Various factor influenced the selection of these color descriptors, these include: Their ability to characterize the perceptual color similarity, judged by performance of the descriptors in matching images and video segments based on color characteristics. Low complexity of the associated extraction and matching techniques. The sizes of the coded descriptions, which play an important role in indexing and in transmission of the descriptors over bandwidth-limited networks. The scalability and interoperability of the descriptors.

Introduction Visual Descriptors Color Descriptors Dominant Color Scalable Color Color Layout Color Structure GoF/GoP Color Texture Descriptors Homogeneous Texture Texture Browsing Edge Histogram Shape Descriptors Region Shape Contour Shape 3D Shape (Normative, basic, for localization) Localization Region Locator Spatio-Temporal Locator Motion Descriptors for Video Camera Motion Motion Trajectory Parametric Motion Motion Activity Other Face Recognition

Type the Color descriptors The Color Space Descriptor: Allows a selection of a color space to be used in the description. The associated Color Quantization Descriptor specifies the partitiong of the given color space into discrete bins. These two descriptors are used in conjunction with other color descriptors. The Dominant Color Descriptors: Allows specification of a small number of dominant color values as well as their statistical properties such as distribution and variance. Its purpose is to provide an effective, compact and intuitive representation of colors present in a region or image. The Scalable Color Descriptors: Is derived from a color histogram defined in the Hue-Saturation-Value color with fixed color space quantization. It uses a Hear transform coefficient encoding, allowing scalable representation of description, as well as complexity scalability of feature extraction and matching procedures.

The Group of Frames or Group of Pictures Descriptor: Is an extension of the scalable Color Descriptors to a group of frames in a video or a collection of pictures. This descriptors is based on aggregating the color properties of the individual images or video frames. The Color Structure Descriptor: Is also based on color histograms, but aims at identifying localized color distributions using a small structuring window. To guarantee interoperability, the color structure descriptor is bound to the HMMD color space. The Color Layout Descriptor: Captures the spatial layout of the representative colors on a grid superimposed on a region or image. Representation is based on coefficients of the discrete Cosine Transform. This is a very compact descriptor being highly efficient in fast browsing and search applications. It can be applied to still images as well we to video segments.

Color Descriptors Color Descriptors Dominant Color Scalable Color - HSV space Color Structure -HMMD space Color Layout -YCbCr space • Color Space: - R, G, B - Y, Cr, Cb - H, S, V - Monochrome - Linear transformation of R, G, B - HMMD GroupOfFrames/ Pictures Constrained color spaces: ->Scalable Color Descriptor uses HSV ->Color Structure Descriptor uses HMMD

Color Space: The color Space Descriptors specifies a selection of a color space to be used in another color descriptor, specifically, the dominant color descriptor. The color spaces specified in the MPEG-7 are RGB, YCbCr, HSV, HMMD, Monochrome and Linear transformation matrix with reference to RGB. In addition, a flag is provided to indicate reference to a color primary and mapping to a standard reference white value. The Color Space Descriptor defines the color components as continuous-value entities. For discrete representation, a quantization is necessary. The color Quantization descriptor specifies the number of quantization levels for each color component in the color space. A uniform quantization in each of the color components in a given color space is assumed.

The RGB color space is one of the more popular color models The RGB color space is one of the more popular color models. This space is defined as the unit cube in the Cartesian coordinate system. The YCbCr is a legacy color space of the precedent MPEG standards, MPEG-1/2/4. It is defined by a linear transformation of RGB color space as follows: Y = 0.299*R + 0.587*G + 0.114*B Cb = -0.169*R – 0.331*G + 0.500*B Cr = 0.500*R – 0.419*G – 0.081*B For the Monochrome color representation, Y component alone in the YCrCb is used.

Scalable Color Descriptor (CSD) A color histogram in HSV color space Encoded by Haar Transform Feature vector: {NoCoef, NoBD, Coeff[..], CoeffSign[..]}

SCD extraction to 11bits/bin to 4bits/bin Nbits/bin (#bin<256)

GoF/GoP Color Descriptor Extends Scalable Color Descriptor for a video segment or a group of pictures (joint color hist. is then possessed as CSD- Haar transform encoding) Extraction Histograms Aggregation methods: Average ..but sensitivity to outliers (lighting changes, occlusion, text overlays) Median ..increased comp. complexity for sorting Intersection ..differs: a “least common” color trait viewpoint

GoF/GoP Color Descriptor Applications: Browsing a large collection of images to find similar images - > Use Histogram Intersection as a color similarity measure for clustering a collection of images - > Represent each cluster by GoP descriptor

Dominant Color Descriptor (DCD) Clustering colors into a small number of representative colors (salient colors) F = { {ci, pi, vi}, s} ci : Representative colors pi : Their percentages in the region vi : Color variances s : Spatial coherency

DCD Extraction (based on Lloyd gen. algorithm) +spatial coherency: Average number of connecting pixels of a dominant color using 3x3 masking window ci centroid of cluster ; x(n) color vector at pixel; v(n) perceptual weight for pixel . H.V.P more sensitive to smooth regions

Color Layout Descriptor (CLD) Clustering the image into 64 (8x8) blocks Deriving the average color of each block (or using DCD) Applying (8x8)DCT and encoding Efficient for Sketch-based image retrieval Content Filtering using image indexing . . . . … . … . . .

CLD extraction > derived average colors are transformed into a series of coefficients by performing DCT (data in time domain - > data in frequency domain). > A few low-frequency coefficients are selected using zigzag scanning and quantized to form a CLD (large quantization step in quantizing AC coef / small quantization step in quantizing DC ). ->The color space adopted for CLD is YCrCb. If the time domain data is smooth (with little variation in data) then frequency domain data will make low frequency data larger and high frequency data smaller. F ={CoefPattern, YDCCoef,CbDCCoef,CrDCCoef,YACCoef, CbACCoef, CrACCoef}

Color Structure Descriptor (CSD) Scanning the image by an 8x8 struct. element Counting the number of blocks containing each color Generating a color histogram (HMMD/4CSQ operating points) It takes into account the colors in the local neighborhood of pixels instead of considering each pixel separately.

CSD extraction F = {colQuant, Values[m]} If Then sub sampling factor p is given by: F = {colQuant, Values[m]}

CSD scaling

The End