CH 11 Multimedia IR: Models and Languages Presented by: Mohammed Al-Rashidi Supervised by: Dr. Mourad Ykhlef

Multimedia IR system The architecture of a Multimedia IR system depends on two main factors The peculiar characteristics of multimedia data The kinds of operations to be performed on such data

Multimedia IR system Support variety of data Different kinds of media Text, images (both still and moving), graphs, and sound Mix of structured and unstructured data Metadata Semi-structured data Data whose structure may not match, or only partially match, the structure prescribed by the data schema The system must typically extract some features from the multimedia objects

Multimedia IR system Data retrieval Exploiting data attributes and the content of multimedia objects Basic steps Query specification Fuzzy predicates (Find all images similar to a car) Content-based predicates (Find all objects containing an apple) Object attributes (Find all red images) Structural predicates (Find all multimedia containing a video clip).

Multimedia IR system Combine DBMS and IR technology 2. Query processing and optimization Query is parsed and compiled into an internal form 3. Query answer 4. Query iteration Combine DBMS and IR technology DBMS: data modeling capabilities IR system: similarity-based query capabilities

Data modeling Main tasks A data model should be defined by which the user can specify the data to be stored into the system Support conventional and multimedia data types Provide methods to analyze, retrieve, and query such data Provide a model for the internal representation of multimedia data

Object-oriented DBMS Provide rich data model Class Drawback More suitable for modeling both multimedia data types and their semantic relationships Class Attributes +operations Inheritance Drawback The performances of storage techniques, query processing, and transaction management is not comparable to that of relational DBMSs Highly non-standard

Object-relational DBMS Extend the relational model Represent complex data types Maintain the performance and the simplicity of relational DBMSs and related query languages Define abstract data types Allows one to define ad hoc data types for multimedia data

Internal representation Using attributes is not sufficient Feature Information extracted from objects Multimedia object is represented as a set of features Features can be assigned manually, automatically, or using a hybrid approach

Internal representation Values of some specific features are assigned to a object by comparing the object with some previously classified objects Feature extraction cannot be precise A weight is usually assigned to each feature value representing the uncertainty of assigning such a value to that feature 80% sure that a shape is a square

SQL3 Support extensible type system Collection data types Provide constructs to define user-dependent abstract data types, in an object-oriented like manner Collection data types Sets, multisets, and lists The elements of a collection must have compatible types

MULTOS MULTimedia Office Server Client/server Support filing and retrieval of multimedia objects Each document is described by a logical structure, a layout structure, and a conceptual structure Documents having similar conceptual structures are grouped into conceptual types

Conceptual structure of the type Generic_Letter spring component type Document Place Date Receiver+ Sender Address Name City Street Country Letter_body Conceptual structure of the type Generic_Letter

Product_Presentation Document Place Date Receiver+ Sender Address Name City Street Country Company_Logo Image Product_Presentation Text Product_Description Product_Cose Signature Letter_body Complete conceptual structure of the type Business_Product_Letter

Query languages Relational/object-oriented database system Exact match of the values of attributes Multimedia IR system Similarity-based approach Considers the structure and the content of the objects Content-based query Retrieve multimedia objects depending on their globe content

Query languages In designing a multimedia query language, three main aspects require attention How the user enters their request to the system Which conditions on multimedia objects can be specified in the user request How uncertainty, proximity, and weights impact the design of the query language

Request specification Interfaces Browsing and navigation Specifying the conditions the objects of interest must satisfy, by means of queries Queries can be specified in two different ways Using a specific query language Query by example Using actual data (object example)

Search Using Sketch in QBIC Sketch entry Results of search

Search by Color in QBIC Color selection 15% yellow and 13% blue Results of search

Search by User Drawn Pattern in QBIC Results of search

Conditions on multimedia data 1. Attribute predicates Concern the attributes for which an exact value is supplied for each object Exact-match retrieval 2. Structural predicates Concern the structure of multimedia objects Can be answered by metadata and information about the database schema “Find all multimedia objects containing at least one image and a video clip”

Conditions on multimedia data 3. Semantic predicates Concern the semantic content of the required data, depending on the features that have been extracted and stored for each multimedia object “Find all the red houses” Exact match cannot be applied

Thanks Any questions