1. Multimedia Content-Based Retrieval
Institute Of Engineering &Technology
Multimedia Content-Based Retrieval
Submitted To:
Mohit Khandelwal Project In-charge CS & IT Department
Presented by:
Sourabh Taletiya
I.T. 4th year [VIIIth sem]

2. Introduction
Multimedia Content-based Retrieval, a technique which uses media contents viz. image ,audio or videos to search multimedia object from large scale multimedia databases according to user’s interests.
An important research area.
Challenging problem since multimedia data needs detailed interpretation from pixel values.
Different strategies in terms of syntactic and semantic indexing for retrieval is used.

3. Why do we need MCBR ?
How do I find what I’m looking for?!

4. Multimedia Content Based Retrieval
Multimedia and Storage Technology that has led to building of a large repository of digital image, video, and audio data.
Compared to text search, any assignment of text labels a massively labour intensive effort.
Focus is on calculating statistics which can be approximately correlated to the content features without costly human interaction.

5. Country: India

7. Multimedia Content-Based Retrieval
Search based on Syntactic features
Shape, texture, color histogram
Relatively undemanding
Search based on Semantic features
Human perception
“ List all dogs look like cat”
“City” “Landscape” “cricket”

8. Syntactic indexing
Use syntactic features as the basis for matching and employ either Query-through-dialog or Query by-example box to interface with the user.
Query-through-dialog
Enter the words describing the image
Query-through-dialog not convenient as the user needs to know the exact details of the attributes like shape, color, texture etc.

9. Image descriptors – Color
Apples are red …
… But tomatoes are too!!!

10. Image descriptors – Texture
Texture differentiates between a Lawn and a Forest

11. Syntactic indexing Query by example
example images and user chose the closest.
various features like color, shape, textures and spatial distribution ,if the chosen image are evaluated and matched against the images in the database.
Similarity or distance metric.
In Video, various key frames of video clips which are close to the user query are shown.

12. Syntactic indexing Query by example limitations
Image can be annotated and interpreted in many ways. For example, a particular user may be interested in a waterfall, another may be interested in mountain and yet another in the sky, although all of them may be present in the same image.
User may wonder "why do these two images look similar?" or "what specific parts of these images are contributing to the similarity?“. User is required to know the search structure and other details for efficiently searching the database.
It requires many comparisons and results may be too many depending on threshold.

13. Semantic indexing Match the human perception and cognition
Semantic content contains high-level concepts such as objects and events.
As humans think in term of events and remember different events and objects after watching video, these high-level concepts are the most important cues in content-based retrieval. Let’s take as an example a cricket game, humans usually remember wickets, interesting actions, innings etc.

14. Semantic indexing
There exists a relationship between the degree of action and the structure of visual patterns that constitute a movie.
Movies can be classified into four broad categories: Comedies, Action, Dramas, or Horror films. Inspired by cinematic principles, four computable video features (average shot length, color variance, motion content and lighting key) are combined in a framework to provide a mapping to these four high-level semantic classes.

15. Matching techniques
Method of finding similarity between the two sets of multimedia data, which can either be images or videos.
Search based on features like location, colors and concepts, examples of which are ‘mostly red’, ‘sunset’, ‘yellow flowers’ etc.
User specify the relative weights to the features or assign equal weightage
Automatically identifying the relevance of the features is under active research.

16. Learning methods in retrieval
The user generates both the positive and negative retrieval examples (relevance feedback).
Each image can represent multiple concepts. To replace one of these ambiguities, each image is modeled as a bag of instances (sub-blocks in the image).
A bag is labeled as a positive example of a concept, if there exist some instances representing the concept, which could be a car or a waterfall scene. If there does not exist any instance, the bag is labelled as a negative example.
The concept is learned by using a small collection of positive and negative examples and this is used to retrieve images containing a similar concept from the database.

17. Future of CBR systems
There is ambiguity in making such conclusions, for example, dissolve can be either due to ‘flashback’ or due to ‘time lapse’. if the number of dissolves is two, most probably ‘flashback’
- “Multimedia Content Description Interface” - specify a standard set of descriptors that can be used to describe various types of multimedia information
Make collaborative effort to tag the multimedia
[2]
[4]
[3]

18. [1]

19. Commercial systems – Like.com

20. Commercial systems – Like.com

21. Commercial systems – Like.com

22. Commercial systems – Like.com

23. Conclusions
Systematic exploration of construction of high-level indexes is lacking.
None of the work has considered exploring features close to the human perception.
In summary, there is a great need to extract semantic indices for making the CBR system serviceable to the user. Though extracting all such indices might not be possible, there is a great scope for furnishing the semantic indices with a certain well-established structure.

24. Conclusions
Content-based video indexing and retrieval is an active area of research with continuing attributions from several domain including image processing, computer vision, database system and artificial intelligence.

25. Refrences WEB [1] http://www.google.com/mobile/goggles
[2] 7.htm
[3]
[4] based_image_retrieval