Presentation is loading. Please wait.

Presentation is loading. Please wait.

Special Topics in Computer Science Advanced Topics in Information Retrieval Lecture 5 (book chapter 11) : Multimedia IR: Models and Languages Alexander.

Similar presentations

Presentation on theme: "Special Topics in Computer Science Advanced Topics in Information Retrieval Lecture 5 (book chapter 11) : Multimedia IR: Models and Languages Alexander."— Presentation transcript:

1 Special Topics in Computer Science Advanced Topics in Information Retrieval Lecture 5 (book chapter 11) : Multimedia IR: Models and Languages Alexander Gelbukh

2 2 Previous Chapter: Conclusions Inverted files seem to be the best option Other structures are good for specific cases oGenetic databases Sequential searching is an integral part of many indexing-based search techniques oMany methods to improve sequential searching Compression can be integrated with search

3 3 Previous Chapter: Research topics Perhaps, new details in integration of compression and search Linguistic indexing: allowing linguistic variations oSearch in plural or only singular oSearch with or without synonyms

4 4 Motivation Applications: ooffice, oCAD, omedical, oInternet Example: oArtists sings a melody and sees all the songs with similar melody

5 5 Whats different Different from text IR: oStructure of data is more complex. Efficiency is an issue oUsing of metadata oCharacteristics of multimedia data oOperations to be performed Aspects: oData modeling: Extract and maintain the features of objects oData retrieval: based not only on description but on content

6 6 Retrieval process Query specification ofuzzy predicates: similar to ocontent predicates: images containing an apple odata type predicates: video,... Query processing and optimization oParsed, compiled, optimized for order of execution oProblem: many data types, different processing for each Answer oRelevance: similarity to query Iteration oBad quality, so need to refine

7 Modeling

8 8 Data modeling To model is to simplify, in order to make manageable. We will represent an image as... oFrom the users point of view oFrom the systems point of view (technically) A problem: very large storage size. Modeling needed Objects are represented as feature vectors oImages / Video: shape. House, car,... oSound: style. Music: Merry, sad,... Features are defined directly or by comparison oDegree of certainty is stored

9 9 Multimedia support in commercial DBMSs Multimedia support in commercial DBMSs (1999) Variable length data. oNon-standard oDifferent and usually very limited sets of operations SQL3: oprovides user-extensible data types oObject-oriented oImplemented partially in many systems Example: data blades of Informix oContent-based functions on text and images oE.g.: date = 1997 AND contains (car)

10 10 Spatial data types Informix: 2D, 3D data blades Boxes, vectors,... Operations: intersect, contains, center,... Text: containWords,.... Supports query images by content

11 11 Example: MULTOS Multimedia document server Documents are described by: ological structure: title, into, chapter,... olayout structure: pages, frames,... oconceptual structure: allows content-based queries oDocs similar in conceptual structures are grouped into conceptual types oExample: Generic_Letter

12 12 Example of conceptual structure...

13 13...continued

14 14 Image data in MULTOS Analysis olow level: detect objects and positions ohigh level: image interpretation Result of analysis: odescription of objects found and their classes ocertainty values Indices are used for fast access to this info oObject index. Includes pointers to objects and certainty values oCluster index, with fuzzy clusters of similar images

15 15 Internet How Google does it? No image processing. Textual context! File names, nearby words Distance from image to words give me images with flower in the file name or near the image

16 Languages

17 17 Query languages As a query, either a description of the object or an example object is submitted oshow me images similar to this one oin what respects similar?! Exact match is inadequate. Additional means are needed Content is not a single feature

18 18 What defines query language Interface. How to enter the query Types of conditions to specify Handling of uncertainty, proximity, weights

19 19 Interface Browsing and navigation Search: description or query by example Query by example: ospecify what features are important. Give me all houses with similar shape but different colors oLibraries of examples can be provided

20 20 Conditions... Attribute predicates ostructured content – the predefined types extracted beforehand oExact match. E.g.: size, type (video, audio,...) Structural predicates ostructure: title, sections,... ometadata are used. Find objects containing an image and a video clip Semantic predicates ounrestricted content. oFind all red houses: red = ?, house = ? Fuzzy

21 21... conditions Predicates oSpatial: contain, intersect, is contained in, is adjacent to... oTemporal: Find audio where first politics and then economy is discussed oSpatial and temporal predicates can be combined: Find clips where the logo disappears and then a graph appears at the same place A predicate can be applied to a part of document oAs path expressions in OO databases

22 22 Uncertainty, proximity, weights Similarity function The user can assign importance weights to individual predicates in a complex query This gives ranking, as in text IR The same models can be used, e.g., probabilistic model

23 23 Examples of query languages: SQL3 Functions and stored procedures: user-defined data manipulation Active database support: database reacts on the events, not only commands. This enforces integrity constraints Good news: rather standard Bad news: no ranking supported! Effort to integrate SQL3 with IR techniques. SQL MM Full Text and other similar languages

24 24... examples: MULTOS One of design goals: easy navigation oPaths are supported Identification of components by type, not by position oAll images in the document, not the image in 3 rd chapter Types of predicates: oon data attributes, on textual components, on images (image type, objects contained,...) Example:

25 25 MULTOS example

26 26 Another example of MULTOS

27 27 Research topics How similarity function can be defined? What features of images (video, sound) there are? How to better specify the importance of individual features? (Give me similar houses: similar = size? color? strructure? Architectural style?) How to determine the objects in an image? Integration with DBMSs and SQL for fast access and rich semantics oIntegration with XML oRanking: by similarity, taking into account history, profile

28 28 Conclusions Basically, images are handled as text described them oNamely, feature vectors (or feature hierarchies) oContext can be used when available to determine features Also, queries by example are common From the point of view of DBMS, integration with IR and multimedia-specific techniques is needed oObject-oriented technology is adequate

29 29 Thank you! Till ??, 6 pm

Download ppt "Special Topics in Computer Science Advanced Topics in Information Retrieval Lecture 5 (book chapter 11) : Multimedia IR: Models and Languages Alexander."

Similar presentations

Ads by Google