1. Selected Advanced Topics
Storing and Retrieving Images
Content-based Image/Video Indexing and Retrieval
QBIC - Query By Image (and video) Content, is an effective technique for visual (image and video) database management.
Various other names are often used in literature, e.g., content-based image indexing and retrieval (CBIR), to refer the same thing.
QBIC is currently an active research topic in computer science and related disciplines. It will play an increasingly important role in multimedia computing, the Internet and general visual data management.

2. Problem
Image/Video Database
Find all images contain horses …..

3. Text-based technology
Annotation: Each image is indexed with a set of relevant text phrases, e.g.,
Retrieval: based on text search technology
Appropriate phrases to describe the content of this image include:
Mother, Child, Vegetable, Yellow, Green, Purple ….

4. Text-based technology - Drawbacks
Annotation - subjective
different people may use different phrases to describe the same or very similar image/content

5. Text-based technology - Drawbacks
Annotation - Laborious
It will take a lot of man-hours to label large image/video databases with 1m+ items

6. Content-based Technology
Using Visual Examples
Image/Video Database

7. Content-based Technology
Using Visual Features r% g% b%
Image/Video Database

8. Content-based Technology
Content-based image indexing and retrieval (CBIR), is an image database management technique, which indexes the data items (images, or video clips) using visual features (e.g., color, shape, and texture) of the images or video clips.
A CBIR system lets users find pictorial information in large image and video databases based on visual cues, such as colour, shape, texture, and sketches.

9. Content-based Technology
The visual features, computed using image processing and computer vision techniques are used to represent the image contents numerically.
Image Content - a high level concept, e.g., this image is a sunset scene, a landscape scene, etc.
Numerical Content Representations - Low level numbers, often the same set of numbers can come from very different images, making the task very hard!

10. Content-based Technology
Techniques for Computing Visual Features/Representing Image Contents –
some are very sophisticated, and many are still not matured
hence the computational processes
in some cases are automatic
but in other cases are semi-automatic
in the most difficult cases, it may have to be done manually

11. Content-based Technology
Comparing Image Content/Retrieving Images based on Content
Simple approaches - compute the metric distance between low level numerical representations
Advanced Approaches - using sophisticated pattern recognition, artificial intelligence, neural networks, and interactive (relevant feed-back) techniques to compare the visual content (low level numerical features)

12. Content-based Technology - IBM QBIC System
The IBM’s QBIC (Query by Image and Video Content) system is one of the early examples of CBIR system developed in 1990s.
The system lets users find pictorial information in large image and video databases based on color, shape, texture, and sketches.

13. Content-based Technology - IBM QBIC System
The User Interfaces Module
Let user specify visual query by drawing, selecting from a color wheel, selecting a sample image …
Display results as an ordered set of images
The Database Population and Database Query Modules
Database population - process images and video to extract features describing their content - colors, textures, shapes and camera and object motion, and store the features in a database
Database Query - let user compose a query graphically, extract features from the graphical query, input to a matching engine that finds images or video clips with similar features

14. Content-based Technology - IBM QBIC System
The Data Model
Still image, or scene - full image
Objects contained in the full image - subsets of an image
Videos - broken into clips called shots - sets of contiguous frames
Representative frames, the r-frames, are generated for each shot
R-frames are treated as still image - from which features are extracted and stored in the database.
Further processing of shots generates motion objects - e.g., a car moving across the screen.

15. Content-based Technology - IBM QBIC System
Queries are allowed on
Objects - e.g., Find images with a red round object
Scenes - e.g., Find images that have approximately 30% red and 15% blue colors
Shots - e.g., Find all shots panning from left to right
A combination of above - e.g., Find images that have 30% red and contain a blue textured objects

16. Content-based Technology - IBM QBIC System
Similarity Measures
Similarity queries are done against the database of pre-computed features using distance functions between the features
Examples include, Euclidean distance, City-block distance, ….
These distance functions are intended to mimic human perception to approximate a perceptual ordering of the database
But, it is often the case that a distance metric in a feature space will bear little relevance to perceptual similarity.

17. Content-based Technology - Basic Architecture
Similarity Measures
Meta data
Imagery
color
texture
shape
positions ….
Record1
color
texture
shape
positions ….
color
texture
shape
positions ….
Record2
color
texture
shape
positions ….
Record3
color
texture
shape
positions ….
Record4
color
texture
shape
positions ….
Record n
Query
Image Database

18. Colour - An effective Visual Cue
Colors can be a more powerful visual cue than you initially thought!
What soft drink
Which fruit?

19. Colour - An effective Visual Cue
In many cases, color can be very effective.
Here is an example
Results of content-based image retrieval using 4096-bin color histograms

20. Colour Spaces Colour Models Sometimes in Computer Vision, it is
RGB Model: This colour model uses the three NTSC primary colours to describe a colour within a colour image. R G B
Yellow
Cyan
Magenta
White
gray
Sometimes in Computer Vision, it is
convenient to use rg chromaticity space
r = R/(R+G+B)
g= G/(R+G+B)

21. Colour Spaces YCbCr Model Y = 0.299R + 0.587G + 0.114B
YIQ Model: The YIQ models is used in commercial colour TV broadcasting, which is a re-coding of RGB for transmission efficiency and for maintaining compatibility with monochrome TV standard.
In YIQ, the luminance (Y) and colour information (I and Q) are de-coupled.
YCbCr Model
Y = 0.299R G B
Cb = R G B
Cr = 0.500R G B

22. Perceived Color Differences
One problem with the RGB colour system is that colorimetric distances between the individual colours don't correspond to perceived colour differences.
For example, in the chromaticity diagram, a difference between green and greenish-yellow is relatively large, whereas the distance distinguishing blue and red is quite small.
r = R/(R+G+B)
g= G/(R+G+B)

23. CIELAB
CIE (Commission Internationale de l'Eclairage) solved this problem in with the development of the Lab colour space. A three-dimensional color space was the result. In this model, the color differences which you perceive correspond to distances when measured colorimetrically. The a axis extends from green (-a) to red (+a) and the b axis from blue (-b) to yellow (+b). The brightness (L) increases from the bottom to the top of the three- dimensional model.
With CIELAB what you see is what you get (in theory at least).

24. Colour Histogram
Given a discrete colour space defined by some colour axes (e.g., red, green, blue), the colour histogram is obtained by discretizing the image colours and counting the number of times each discrete colour occurs in the image.
The image colours that are transformed to a common discrete colour are usefully thought of as being in the same 3D histogram bin centered at that colour.

25. Colour Histogram Construction
Step 1
Colour quantization (discretizing the image colours)
Step 2
Count the number of times each discrete colour occurs in the image.

26. Colour Quantization
A true colour, 24-bit/pixel image (8 bit - R, 8 bit - G, 8 bit -B), will have 224 = bins !
That is, each image will have to be represented by over 16 million numbers
computationally impossible
in practice not necessary
Colour quantization - reduce the number of (colours) bins

27. Simple Colour Quantization
Simple Colour Quantization (Non-adaptive)
Divide each colour axis into equal length sections (different axis can be divided differently).
Map (quantize) each colour into its corresponding bin

28. Simple Colour Quantization
Example: In RGB space, quantize each image colour into one of 8x8x8 = 512 colour bins 31 63 95
127
159
191
223
255 R G B
Colour Bin Colour Bin
(123,23,45) (3, 0, 1 ) (122, 28, 46) (3, 0, 2)
(132, 29,50) (4, 0, 1) (122, 172, 27) (3, 5, 0)
(121,26,48) (x, x, x) (142, 28, 46) (x, x, x)

29. Advanced Colour Quantization
Adaptive Colour quantization (Not required)
Vector Quantization
K-means clustering
K representative colours
The colour histogram consists of K bins, each corresponding to one of the representative colours.
A pixels is classified as belonging to the nth bin if the nth representative colour is the one (amongst all the representative colours) that is closest to the pixel.
A pixel is a point in the 3D colour space B G R
Representative colours

30. Colour Histogram Construction - An Example
A 3 x 3, 24-bit/pixel image has following RGB planes
Construct an 8-bin colour histogram (using simple colour quantization, treating each axis as equally important).
Red
Green
Blue
Bin (0,0,0) = Bin (0,0,1) = Bin (0,1,0) = Bin (0,1,1) =
Bin (1,0,0) = Bin (1,0,1) = Bin (1,1,0) = Bin (1,1,1) =

31. Colour Histogram Construction - An Example
Quantized Colour Planes
Count the number of times each discrete colour occurs in the image.
Red
0 0 0
Green
1 0 0
0 1 1
0 0 0
Blue
0 0 0
Bin (0,0,0) = 6 Bin (0,0,1) = 0 Bin (0,1,0) = 3 Bin (0,1,1) = 0
Bin (1,0,0) = 0 Bin (1,0,1) = 0 Bin (1,1,0) = 0 Bin (1,1,1) = 0

32. Colour Based Image Indexing
The histogram of colours in an image defines the image colour distribution
# of pixels
# of pixels 10
100 30
Color
Distribution =
(10,0,0,0,100,10,30,0,0)

33. Colour based Image Retrieval
Images are similar if their histograms are similar!
Colour
Distribution =
(10,0,0,0,100,10,30,0,0)
Dissimilar
Colour
Distribution
Similar! =
(0,40,0,0,0,0,0,0,110,0)
Colour
Distribution =
(10,0,0,0,90,10,40,0,0)

34. Formalizing Similarity
Colour
Distribution =
(10,0,0,0,100,10,30,0,0) 1 2
Colour
Distribution =
(0,40,0,0,0,0,0,110,0)
Similarity(Image 1, Image 2) = D (H1, H2)
where D( ) is a distance measure between vectors (histograms) H1 and H2

35. Metric Distances
A distance measure D( ) is a good measure if it is a metric!
D(a,b) is a metric if
D(a,a) = 0 (the distance from a to itself is 0
D(a,b) = D(b,a) (the distance from a to b = distance from b to a)
D(a,c) <= D(a,b) + D(b,c)
( triangle inequality [ the straight line distance is always the least!] ) a b c
D(a,b) + D(b,c) should be no smaller
than D(a,c)

36. Common Metric Distance measures
Histogram Intersection, HI
HI(H1, H2) =
H1 = (10, 0, 0, 0, 100, 10, 30, 0, 0)
H2 = ( 0, 40, 0, 0, 0, , 0, 110, 0)
Similarity = HI(H1, H2) = = 6

37. Common Metric Distance measures
Euclidean or straight-line distance or L2-norm, D2
Root-mean square
error
H1 = (10, 0, 0)
H2 = ( 0, 40, 0)
Similarity = D2(H1, H2) = sqrt( ) = 41.23

38. Common Metric Distance measures
Manhattan or city-block or L1-norm, D1
sum of absolute
differences
H1 = (10, 0, 0)
H2 = ( 0, 40, 0)
Similarity = D1(H1, H2) = ( ) = 50

39. Histogram Intersection vs City Block Distance
Theorem: if H1 and H2 are colour histograms and the total count
in each is N (there are N-pixels in an image) then:
(Histogram Intersection inversely proportional to a metric distance!)
Proof
(by definition)
(1)
(2)

40. Histogram Intersection vs City Block Distance
(3)
Substituting (2) and (3) in (1)
(4)
(5)

41. Colour Histogram Database
Build
Histogram
Database
(1)
(2)

42. How well does Color histogram intersection work ?
66 test histograms in the database
Swain Original Test:
31 query images
Recognition rate almost 100%
Indeed, because color indexing worked so well it is at
he heart of almost all image database systems

43. Google Image Search

44. Google Image Search
After clicking this colour patch

45. Problems with color histogram matching
1. Color Quantization problem:
Colour
Distribution =
(0,40,0,0,0,0,0,110,0)
Because, the
two images have
slightly different
color distributions
their histograms have
nothing in common!
0 intersection!
Colour
Distribution =
(0,0,40,0,0,0,0,0,110)
Sources of quantization error: noise, illumination, camera

46. Problems with color histogram matching
2. The resolution of a color histogram
Colour
Distribution =
(0,40,0,0,0 … ,0,0,110,0)
For the best results, Swain quantized colour
space into 4096 distinct colours => Each colour
distribution is a 4096-dimensional vector.
=> Histogram intersection costs O(4096) operations
(some constant * 4096)
4096 comparisons per database histogram => histogram intersection
will be very slow for large databases
Many newer methods work well using D features

47. Problems with color histogram matching
3. The colour of the light
Under a yellowish light all image colours are
more yellow than they ought to be

48. Problems with color histogram matching
4. The structure of colour distribution
All four images have the same color distribution - need to take into account spatial relationships!

49. Problem solution => Use statistical moments
1st order statistics =
Mean
2nd order statistics =
Variance/
Covariance

50. Statistical similarity
Colour
Distribution =
(50,50,50)
Compare mean RGBs
(In general compare all statistical measures)
Colour
Distribution =
(20,70,40)
Statistical similarity =
(Euclidean distance between corresponding statistical measures)

51. Histogram vs Statistical Similarity
Completeness of
representation
#params/
Match speed
Sensitivity to Quantization error
histogram
complete
Many/slow
sensitive
Low order stats
incomplete
Few/fast
insensitive
Low and high
order stats
complete
(or over complete)
Many/moderate
sensitive

52. Advanced Topics Fast Indexing Interactive/Relevant Feedback
Reducing the Semantic Gap
Visualization, Navigation, Browsing
Internet scale image/video retrieval
Flickr – billions of photos
Youtube – billions of videos …