1. BIT 3193 MULTIMEDIA DATABASE CHAPTER 4 : QUERING MULTIMEDIA DATABASES

3. The structure of image is much less explicit. so need to apply techniques that will identify a structure characterizing the content of visual objects is much more complex and uncertain. characterized by feature vectors A feature is an attribute derived from transforming the original visual object by using an image analysis algorithm.

4. The visual query mode involves matching the input image to pre-extracted features of real objects. pre-extracted features are held in the database Purpose: to extract a set of numerical features that removes redundancy from the image and reduces its dimension The most commonly used features for content-based image retrieval are shape, color and texture.

5. A content-based image retrieval (CBIR) system uses image visual content features to retrieve relevant images from an image database. CBIR systems retrieve images according to specified features that users are interested in. features such as texture, color, shape, and location properties can reflect the contents of an image

6. Example: Trainable System for Object Detection A set of positive example images of the object class considered (e.g., images of frontal faces) and a set of negative examples (e.g., any non face image) are collected. The images are transformed into (feature) vectors in a chosen representation (e.g., a vector of the size of the image with the values at each pixel location below this is called the “pixel” representation)

7. The vectors (examples) are used to train a pattern classifier, the Support Vector Machine (SVM), to learn the classification task of separating positive from negative examples. To detect objects in out-of-sample images, the system slides a fixed size window over an image and uses the trained classifier to decide which patterns show the objects of interest. At each window position, the system extracts the same set of features as in the training step and feed them into the classifier; the classifier output determines whether or not it is an object of interest.

8. Representation Technique for Face and People Detection Pixel Representation Eigen Vector Representation Wavelet Representation Multimedia data, such as images or video, are typically represented or stored as very high-dimensional vectors. The processing time for searching or performing other operations for such systems is highly impacted by the fact that the data are so high dimensional. It is therefore practically important to find compact representations of multimedia data, while at the same time not significantly affecting the performance of systems such as detection

9. Can be based on: color using color histograms and color variants texture variation in intensity and topography of surfaces shape using aspect ratios circularity and moments for global features using boundary segments for local features

10. Can be based on: position using spatial indexing image transformations using transformations appearance using a combination of color, texture and intensity surfaces

11. FeatureMeasuresTheoryMain useProblems ColorHistogramSwain and BallardColor indexingLighting variations Texture Pixel Intensity -illumination -topography Degrees of -directionality -regularity -periodicity Gabor filters Fractals Indexing Texture thesaurus Shape Global features -aspect ratio -circularity -moments Local features - boundary segments Active contours Shape indexing Object recognition Spikes and holes in objects cause errors in indexing Appearance Global features -curvature -orientation Local features - local curvatures and orientation TransformsImage classification PositionSpatial relationshipsTessellations (Voronoi)Object Recognition Table 4.1 : Features used in retrieval

12. FeatureAdvantageDisadvantage Color Can be applied to all colored images, 2D and 3D Texture Distinguishes between image regions with similar color e.g sea and sky Large feature vectors each containing 4000 elements have been used Shape Important in image segmentation Can classify images as stick like, plate like or blob like Representation is difficult Viewpoint change an object’s shape Spikes and holes 3D is very difficult Appearance Important way of judging similarity Can generate invariant measures Describe an image at varying levels of detail PositionCan be applied to 2D and 3D images Images must contain objects in defined spatial relationships Spatial indexing not useful unless combined with color and texture FeatureAdvantageDisadvantage Color Can be applied to all colored images, 2D and 3D Texture Distinguishes between image regions with similar color e.g sea and sky Large feature vectors each containing 4000 elements have been used Shape Important in image segmentation Can classify images as stick like, plate like or blob like Representation is difficult Viewpoint change an object’s shape Spikes and holes 3D is very difficult Appearance Important way of judging similarity Can generate invariant measures Describe an image at varying levels of detail PositionCan be applied to 2D and 3D images Images must contain objects in defined spatial relationships Spatial indexing not useful unless combined with color and texture Table 4.2 : Advantages and Disadvantages of features methods of retrieval

13. There are two alternative approaches: use a query image user can provide an image or compose a target image by selecting and clicking color palettes and texture patterns use user-defined features allow user to select a sample image query process the distribution of image objects is then recomputed in terms of the distance from sample image

15. use automatic methods for generating content- dependent metadata speech recognition techniques is used for the identification of both speakers and the spoken words factors which influence the complexity of the identification problems encountered include: isolated words (easier to recognize) single speaker (one is easier) vocabulary size (smaller is easier) grammar (tightly constraint is easier)

16. users can use query by example (QBE) the technologies used to achieve this have to be integrated and include: large vocabulary speech recognition speaker segmentation speaker clustering speaker identification name spotting topic classification story segmentation

18. Videos are far more complex. Role of video feature extraction: image-based features motion-based features (e.g motion of the camera) object detection and tracking speech recognition speaker identification word spotting audio classification

19. Clip 1 Story 1Story m Shot 1Shot 2 Shot k Clip Scene / Story Segment Shot captured between a record and stop camera operation

20. Clip digital video document that can last from a few seconds to a few hours Scene sequential collection of shots unified by a common event or locale (background). a clip have one or more scenes

21. Shot fundamental unit much research has focused on segmenting video by detecting boundary between camera shots defined as a sequence of frames captured by a single camera in a single continuous action in time and space example : two people having a conversation low-level syntactic building blocks of a video sequence

22. The video operations are: create concatenate, union and intersection (based on temporal and spatial conditions) output

23. Query example: “ Show the details of movies where a character said “I am not interested in a semantic argument, I just need the protein”

24. Access control and rights management Query Processing Query Presentation IndexesVisual summaries Video processing and annotation summaries Digital video collection Content delivery Query results Query inputs User Figure A : Video retrieval system

25. ISO/IEC 13249 (SQL/MM) SQL Multimedia and Applications Standardized in 2001 by ISO subcommittee SC32 Working Group Provides structured object types, methods to store, manipulate image data by content Supports OR (Object Relational) Data Model  Part 1: Framework  Part 2: Full Text  Part 3: Spatial  Part 5: Still Image  Part 6: Data Mining

27. Object types that comply with the first edition of the ISO/IEC 13249-5:2001 SQL MM Part5: StillImage standard SI_AverageColor Object Type Describes the average color feature of an image. SI_Color Object Type Encapsulates color values of a digitized image. SI_ColorHistogram Object Type Describes the relative frequencies of the colors exhibited by samples of an image.

28. SI_FeatureList Object Type Describes an image that is represented by a composite feature. The composite feature is based on up to four basic image features (SI_AverageColor, SI_ColorHistogram, SI_PositionalColor, and SI_Texture) and their associated feature weights. SI_StillImage Object Type Represents digital images with inherent image characteristics such as height, width, format, and so on.

29. SI_PositionalColor Object Type Describes the positional color feature of an image. Assuming that an image is divided into n by m rectangles, the positional color feature characterizes an image by the n by m most significant colors of the rectangles. SI_Texture Object Type Describes the texture feature of the image characterized by the size of repeating items (coarseness), brightness variations (contrast), and predominant direction (directionality).

30. Read the following website for further information on Oracle implementation of SQL/MM Still Image: http://download.oracle.com/docs/cd/B19306_01/appdev.102/ b14297/ch_stimgref.htm#CHDBAGID.

31. Example of media table for still Images defined as per SQL/MM standards Given the following PM.SI_MEDIA table definition in Oracle implementation: CREATE TABLE PM.SI_MEDIA( PRODUCT_ID NUMBER(6), PRODUCT_PHOTO SI_StillImage, AVERAGE_COLOR SI_AverageColor, COLOR_HISTOGRAM SI_ColorHistogram, FEATURE_LIST SI_FeatureList, POSITIONAL_COLOR SI_PositionalColor, TEXTURE SI_Texture, CONSTRAINT id_pk PRIMARY KEY (PRODUCT_ID));

32. Example1: Construct an SI_AverageColor object from a specified color using the SI_AverageColor(averageColorSpec) constructor. Solution : DECLARE myColor SI_Color; myAvgColor SI_AverageColor; BEGIN myColor := NEW SI_COLOR(null, null, null); myColor.SI_RGBColor(10, 100, 200); myAvgColor := NEW SI_AverageColor(myColor); INSERT INTO PM.SI_MEDIA (product_id, average_color) VALUES (75, myAvgColor); COMMIT; END;

33. Example 2: Derive an SI_AverageColor value using the SI_AverageColor(sourceImage) constructor: Solution : DECLARE myimage SI_StillImage; myAvgColor SI_AverageColor; BEGIN SELECT product_photo INTO myimage FROM PM.SI_MEDIA WHERE product_id=1; myAvgColor := NEW SI_AverageColor(myimage); END;

34. Example 3: Insert into PM.SI_MEDIA table an object with PRODUCT_ID = 1 and have average color of RED = 20, GREEN = 30 and BLUE = 50. Solution : DECLARE myColor SI_Color; myAvgColor SI_AverageColor; BEGIN myColor := NEW SI_COLOR(null, null, null); myColor.SI_RGBColor(20, 30, 50); myAvgColor := NEW SI_AverageColor(myColor); INSERT INTO PM.SI_MEDIA (product_id, average_color) VALUES (1, myAvgColor); COMMIT; END;

35. Example 4: Derive SI_AverageColor object for image with PRODUCT_ID = 13 using the SI_FindAvgClr() function. Solution: DECLARE myimage SI_StillImage; myAvgColor SI_AverageColor; BEGIN SELECT product_photo INTO myimage FROM PM.SI_MEDIA WHERE product_id=13; myAvgColor := SI_FindAvgClr(myimage); END;

37. In 2002, ISO subcommittee MPEG published a standard: MPEG-7 Formally named Multimedia Content Description Interface MPEG-4, the first Multimedia representation Standard Object based coding MPEG-7, Currently the most complete description standard for multimedia data Any audio/visual material associated with multimedia data can be indexed & searched Provides Set of descriptors (D) Quantitative measures of audio/visual features Description Scheme (DS) Structure of Descriptors & relationship

38. MPEG-7 descriptions associated with Still pictures, graphics, 3D models, audio, speech, video Composition information about how these elements are combined in a multimedia presentation (scenarios) MPEG-7 descriptions do not depend on the ways the described content is coded or stored It is possible to create an MPEG-7 description of an analogue movie or of a picture that is printed on paper, in the same way as of digitized content.

39. MPEG-7 can exploit the advantages provided by MPEG-4 coded content Material encoded using MPEG-4 provides the means to encode audio-visual material as Objects having certain relations in time (synchronization) and space (on the screen for video, or in the room for audio), Possible to attach descriptions to elements (objects) within the scene, such as audio and visual objects

40. Same material can be described using different types of features, tuned to the area of application Eg : A visual material: Lower abstraction level would be a description of shape, size, texture, color, movement (trajectory) and position ( “where in the scene can the object be found?”) The highest level would give semantic information: “ This is a scene with a barking brown dog on the left and a blue ball that falls down on the right, with the sound of passing cars in the background”

41. Apart from the description what is depicted in the content, Following additional information about the multimedia data: The form (e.g. JPEG, MPEG-2), The overall data size (helps determining whether the material can be “read” by the user terminal) Conditions for accessing the material (Includes links to a registry with intellectual property rights information, and price) Classification -(Includes parental rating, and content classification into a number of pre-defined categories) Links to other relevant material -(helps the user speeding up the search) The context -( The occasion of the recording, Like Olympic Games 1996, final of 200 meter hurdles, men )

42. Main elements of the MPEG-7 standard Description Tools: Descriptors (D), Description Schemes (DS), A Description Definition Language (DDL) Defines the syntax of the MPEG-7 Description Tools and to allow the creation of new Description Schemes System tools Supports binary coded representation for efficient storage and transmission, transmission mechanisms (both for textual and binary formats), multiplexing of descriptions, synchronization of descriptions with content, management and protection of intellectual property in MPEG-7 descriptions, etc.

43. The key info that the description tools capture includes Structural information on spatial, temporal or spatio-temporal components of the content (scene cuts, segmentation in regions, region motion tracking). Low level features in the content (colors, textures, sound timbres, melody description). Conceptual information of the reality captured by the content (objects and events, interactions among objects). Information about how to browse the content in an efficient way (summaries, variations, spatial and frequency subbands,). Information about collections of objects. Information about the interaction of the user with the content (user preferences, usage history)

44. Scope of MPEG-7

45. MPEG-7 Main Elements

46. Abstract representation of possible applications using MPEG-7

47. Integration of MPEG-7 into MMDBMS MPEG-7 relies on XML Schema, mapping strategies from XML to database data model is an issue!!! SQL/MM, Querying Due the rich description provided by MPEG-7, enhancements in SQL/MM is needed Operations that manipulate, produce as results, an XML is an option Indexing methods for multidimensional data can be used to index multimedia data MPEG-7 Provides methods for semantic indexing!!!

48. More on MPEG-7 can be found from ISO/IEC JTC1/SC29/WG11 CODING OF MOVING PICTURES AND AUDIO- MPEG-7 Overview http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm