1. Multimedia Information Retrieval
Unlike alphanumeric data, multimedia data do not have any semantic structure
Achieving symmetry between annotation and query is difficult
Retrieval is based on similarity between query and stored information instead of exact match
Stored information is represented using indexing

2. IR Model
Information is preprocessed to extract features and semantic contents
Indexed based on these features and semantics
User’s query is processed and main features are extracted
Query’s features are then compared with features or index of each information item in the database
Information item whose features are similar to those of the query are retrieved and presented to the user

3. Design Issues Indexing
a mechanism that reduces the search space of an operator without losing any relevant information
Similarity Computation
easy to compute and should conform to human judgement

4. Performance Measures Retrieval speed, recall, precision
Recall measures the ability of retrieving relevant information items from the database
defined as the ratio between the number of retrieved relevant items and the total number of relevant items in the database
Precision measures retrieval accuracy
defined as the ratio between the number of retrieved relevant items and the number of total retrieved items
Recall and precision are usually considered together
high recall and low precision
high precision and low recall

5. Text Retrieval
Text may be used to annotate other media such as audio, images and video and conventional IR techniques used to retrieve multimedia information
Boolean IR systems or text-pattern search systems
Substantial effort is spent in analyzing the contents of the documents and in generating keywords and indices
Boolean queries are keywords connected with logical operators (AND, OR, NOT)

6. File Structures Flat files Inverted files
for each term a separate index is constructed that stores the document identifiers for all documents containing the term
each term and the document IDs containing the term are organized into one row
searching and retrieval is fast because only rows containing the query terms need to be retrieved and there is no need to search the whole database

7. Extensions
Nearness parameters used in query specification help define the topic more precisely and therefore increase probable relevance of the retrieved item
Within Sentence and Adjacency specification in queries
Term location information is included in the inverted file
Term i : document id, paragraph no., sentence no., word no.
For example, if an inverted file has the following entries:
information: R99, 10, 8, 3; R155, 15, 3, 6; R166, 2, 3,1
retrieval: R77, 9, 7, 2; R99, 10, 8, 4; R166, 10, 2, 5

8. Indexing
Stop words -- grammatical functional words, such as “of,” “the,” and “a.”
Stemming -- reducing words to a common root form
Thesaurus -- list of synonyms
Weighting -- term significance derived from occurrence frequency within a document and among different documents

9. Relevance Feedback Query modification Document modification
terms occurring in documents previously identified as relevant are added to the original query or the weight of such terms is increased
terms occurring in documents previously identified as irrelevant are deleted from the query or the weight of such terms is reduced
Document modification
terms in the query, but not in the user-judged relevant documents, are added to the document index list with an initial weight
weights of index terms in the query and also in relevant documents are increased by a certain amount
weights of index terms not in the query but in the relevant documents are decreased by a certain amount

10. Problems with Annotation
Automatic generation of descriptive key words or extracting semantic information to build classification hierarchies for broad varieties of images
Involving human operators makes the process time-consuming and subjective
retrieval fails if the user forms a query based on key words not employed by the operator
retrieval fails if the query refers to elements of image content that were not described
certain visual properties, textures and shapes, are difficult or nearly impossible to describe with text for general-purpose usage

11. Content-based IR
Retrieve visual data using queries based on the visual content of an image/video : patterns , colors, textures, and shapes, layout and location information
when it is necessary to verify that a trademark or logo has not been used by another comapany
comparing fabric patterns
Search is driven by first establishing one or more sample images and then identifying specific features of those sample images which need to match images from the database

12. Audio Search and Retrieval
Keywords can be highly subjective because of a different perspective or even a different taxonomy
Hard to browse directly since it must be auditioned in real-time (unlike video which can be keyframed)
Two categories : Speech and Non-speech
with speech, indexing and retrieval is based on obtaining spoken words either manually or by speech recognition technique
with non-speech, indexing and retrieval may be based on text annotation (but will it help a query like “find the first occurrence of the note G-sharp.”)

13. Image Database Issues
Selection, derivation, and computation of image features and objects that provide useful query expressiveness
Retrieval methods based on similarity, as opposed to exact matching
User interface that supports the visual expression of queries and allows query refinement and navigation of results
Indexing which is compatible with the expressiveness of the queries
A system architecture that supports this approach

14. Color Analysis
Color distribution represented as a histogram of intensity values each of whose bins corresponds to a range of pixel values
Histograms are compared by an intersection operation.
This sum may be interpreted as enumerating the number of pixels which are common to both histograms
This value may be normalized by the total number of pixels in one of the two histograms
Computationally expensive -- O(NM) where N is th enumber of histogram bins and M is the total number of images in the database

15. Color Analysis (contd.)
Reduce search time by reducing the number of histogram bins
transform RGB representation (coarse segmentation of color space)
apply clustering technique to determine K best colors in a given color space (clustering process takes into account the color distribution of images over the entire database)
a small number of histogram bins tend to capture the majority of pixels of an image; only largest bins in terms of pixels counts need be selected as representation of any histogram. As long as the bins of the query and image histograms are appropriately matched, intersection may be computed over this reduce set.

16. Color Analysis (contd.)
Disadvantages:
histogram-based similarity computation lacks information about location (this problem may be solved by dividing an image into sub-areas and calculating a histogram for each of those sub-areas
image representations in the image database as well as queries have to be the same

17. Texture Analysis
Statistical methods are used to characterize texture in terms of the spatial distribution of image intensity
Tamura features:
contrast : quantification is based on the statistical distribution of pixel intensities
coarseness : measure of the granularity of the texture
directionality : to compute this measure, a gradient vector is calculated at each pixel

18. Shape Analysis Histogram of significant edges
Ordered list of interest points
Chain-code-based shape representation and similarity measure

19. Chain Code-based Shape Analysis
4-directional
8-directional
Grid spacing
Normalization process -- starting point, rotation, scale

20. Starting Point Normalization
Treat the chain code generated by an arbitrary starting point as a circular sequence of direction numbers
Redefine the starting point such that the resulting sequence of numbers forms an integer of minimum magnitude
(arbitrary starting point)
(after normalizing)
After normalizing, the shape boundary has unique chain code (for a fixed orientation and grid size)

21. Shape Number
Rotation normalization is needed because a boundary after rotation has a different chain code. Rotation changes the spatial relationships between the grid space and boundary.
First difference of the chain code reflects spatial relationships between boundary segments which are independent of rotation
The difference is computed by counting (in a counter- clockwise) the number of directions that separate two adjacent elements in a code
Shape number of a boundary is defined as the first difference of the smallest magnitude

22. Unique Shape Number
Need for making the shape boundaries invariant to rotation and scale
Solution -- orient the resampling grid along the principal axis of the shape boundary. In this case, the grid and the boundary have fixed spatial relationships.
Major axis is defined as the line segment between two farthest points on the boundary. Minor axis is perpendicular to the major axis and its length is such that a rectangle formed by these axes will enclose the shape boundary.

23. Scale Normalization
Eccentricity of the boundary -- ratio of the major to the minor axes
Basic Rectangle -- rectangle formed by the major and the minor axes of a boundary
Shape number obtained using basic rectangle will be unique

24. Unique Chain Code Algorithm
select the first digit as any number within the chain code direction range, say 0;
the second digit differs from the first digit by an amount determined by the first digit of the shape number
use the shape number to determine the rest of the digits in the unique chain code

25. Similarity Measurement
The distance d between two boundaries is defined as the number of grids not commonly covered by the two boundaries
boundaries with the same unique chain code have distance 0
Obtain a binary number for each boundary
Exclusive OR of the binary numbers of the two boundaries and the number of 1s in the result is the distance d
Similarity is 1 - (d/N)

26. Indexing and Retrieval of Video
Video is normally made of a number of logic units or segments (video shots)
frames depicting the same scene
frames signify single camera operation
frames contain a distinct event or or action (signifying the presence of the same object)
Consecutive frames on either side of a camera break generally display a significant quantitative change in the content (other camera operations such as dissolve, wipe, fade-in, and fade-out require sophisticated measures to quantify the change)

27. Shot Detection
Difference metrics between frames are based on the comparison of pixel intensity histograms
Difference threshold are chosen such that all boundaries are detected and false detection is minimized
Dealing with gradual changes requires sophisticated techniques
Indexing is done by finding a representative frame and features of this frame are extracted and indexed based on text, color, shape, and/or texture