1. Visual Information Retrieval
Introduction

2. Visual Information Retrieval
Information retrieval, image/video analysis and processing, pattern recognition and computer vision, visual data modeling and representation, multimedia database organization, multidimensional indexing, psychological modeling of user behavior, man-machine interaction and data visualization

3. Visual Information Retrieval
Types of associated information
content-independent metadata (CIM)
format, author's name, date, location, ownership
content-dependent metadata (CDepM)
low-level features concerned with perceptual facts: color, texture, shape, spatial relationship, motion
content-descriptive metadata (CDesM)
high-level content semantics: cloud

4. Visual Information Retrieval
First-generation visual information retrieval systems
CIM by alphanumeric strings, CDepM and CDesM by keywords or scripts

6. Visual Information Retrieval
find images of paintings by Chagall with a blue background
Select IMAGE# from PAINTINGS where PAINTER = "Chagall" and BACKGROUND = "blue"
find images of paintings by Chagall with a girl in red dress and a blue background
full text retrieval

7. Visual Information Retrieval
find images of paintings depicting similar figures in similar positions
it is difficult for text to capture the perceptual saliency of some visual features
text is not well suited for modeling perceptual similarity
perception is mainly subjective, so is its text descriptions

8. Visual Information Retrieval
New-generation visual information retrieval systems
retrieval not only by concepts but also by perception of visual contents
objective measurements of visual contents and appropriate similarity models
automatically extract features from raw data by image processing, pattern recognition, speech analysis and computer vision techniques

9. Visual Information Retrieval
Image retrieval
by perceptual features
for each image in the database, a set of features (model parameters) are precomputed
to query the image database
express the query through visual examples
select features and ranges of features
choose a similarity measure
compute similarity degrees, ranking, relevance feedback

13. Visual Information Retrieval

14. Visual Information Retrieval
system architecture
extraction of perceptual features (CDepM)
extraction of high-level semantics (CDesM) from low- level features
manual annotation of CIM and CDesM
index structure
graphical query tool
retrieval engine
visualization tool
relevance feedback mechanism

15. Visual Information Retrieval
Video retrieval
special characteristics
frames are linked together using editing effects
cut, fade, dissolve
content of the frames
characters, story
changes in color, texture, shape and position (camera or object) in multiple frames
techniques to obtain video streams
different types of video
commercial, news, movies, sport

16. Visual Information Retrieval
by structure
frame: basic unit of information
shot: elementary segment of video with perceptual continuity
clip: set of frames with some semantic meaning
scene: consecutive shots with simultaneous space, time and action
episode: specific sequence of shot types such as a news episode

18. Visual Information Retrieval
by content
perceptual properties, motion and type of an object
situations between objects
motion of camera
semantics of shots by color- or motion-induced sensations
semantics of scenes
stories
audio properties: dialogue, music or storytelling
textual information: caption or text recognized from video

20. Visual Information Retrieval
system architecture
extraction of shots and the associated semantics
key-frames or mosaics
extraction of scenes and stories
manual annotation tool
browsing/visualization tool
video summarization
graphical query tool
index structure
retrieval engine

21. Visual Information Retrieval
3D image and video retrieval
WWW visual information searching
efficiency has to be emphasized due to limited network bandwidth
operate in compressed domain
visual summarization
visualization at different levels of resolution

22. Visual Information Retrieval
Research directions
tools for automatic extraction of low-level features
tools for automatic extraction of high-level semantics
models for representing visual content
at several abstraction levels
effective indexing
effective database models

23. Visual Information Retrieval
visual interfaces
allow querying and browsing
allow querying by text and visual information
similarity models
fit human similarity judgment
psychological similarity models
Web search
content + references
3D image and video retrieval