1. Lecture 12 Content-Based Image Retrieval
Slides by:
Deepak Bote, Xiaoguang Feng,
David A. Forsyth, Clark F. Olson,
Yossi Rubner, Linda G. Shapiro

2. Image databases
Large collections of image (and video) occur in many applications:
Stock photos and footage
Military
World-wide web
Medical imaging
Space exploration
Surveillance
Many others…

3. Image databases
Most collections of images and videos are not image databases.
No DBMS manages the data
No facility for complex queries is available
This is a rapidly expanding area of interest in computer vision:
How can we find a particular image that we are interested in?
Can we locate images that meet some description?
How can we organize the collection meaningfully?
Can we extract new information by exploring the collection automatically?

4. History

5. Image retrieval by annotation
Traditional text-based image search engines
Manual annotation of images
Use text-based retrieval methods
E.g.
Water lilies
Flowers in a pond
<Its biological name>

6. Limitations of annotation
Problem of image annotation
Large volumes of databases
Valid only for one language – with image retrieval this limitation should not exist
Problem of human perception
Subjectivity of human perception
Too much responsibility on the end-user
Problem of deeper (abstract) needs
Queries that cannot be described at all, but tap into the visual features of images

7. Image retrieval
Image have rich content that can be used for retrieval!
The problem that has been examined the most is: How can I find images that meet some description or are like an example?
This is often called “content-based image retrieval” (CBIR). Image content is used, rather than metadata.
Using example images:
Takes the responsibility of forming the query away from the user.
Allows each image to be described by its own features.
Mean color, histogram, etc.

8. Measures No system is perfect. Usually measured using two criteria:
Recall: percentage of correct items found
Precision: percentage of items found that are correct
Which is more important?

9. User interaction
Due to the imperfections in current methods, most systems have a user “in the loop”.
Retrieve-refine-retrieve cycle
Y. Rubner, C. Tomasi, L.J. Guibas, “A metric for distributions with applications to image databases”, Intl. Conf. Computer Vision. © 1998, IEEE.

10. Features for retrieval
What features are used for retrieving images?
Color
- Mean
- Overall distribution
- Relative locations
Texture
- Linear filters
- Textures of textures (!?)
Shape
- Sketches
- Segmented objects
Others

11. Color histograms
An early (and still popular) similarity measurement uses color histograms.
The RGB (or another) color space is discretized into bins.
For each bin, a count is maintained on the number of pixels that fall into the bin (since they have the right color)
Once constructed, the histograms can be compared using several metrics.
UC Berkeley Digital Library Project.

12. Color histograms
The QBIC system (IBM) was the first commercial system created.
It uses color, texture, shape, location, and keywords.
h(I) is a K-bin histogram of a database image
h(Q) is a K-bin histogram of the query image
A is a K x K similarity matrix
The QBIC color histogram distance is:
dhist(I,Q) = (h(I) - h(Q)) A (h(I) - h(Q)) T

13. ** Images courtesy : Yong Rao
QBIC – Search by color
** Images courtesy : Yong Rao

14. ** Images courtesy : Yong Rao
QBIC – Search by shape
** Images courtesy : Yong Rao

15. ** Images courtesy : Yong Rao
QBIC – Query by sketch
** Images courtesy : Yong Rao

16. Histogram distances

17. Histogram distances

18. Earth mover’s distance

19. Earth mover’s distance

20. Layout templates
Color histograms have no information about pixel locations. This yields false positives.
We can add spatial information by using a template of the image.
Example: Blue at top (sky), white below (snowy mountains), dark at bottom (mountains or lake)
Simplest implementation would divide image into regular blocks.
P. Lipson, E. Grimson, P, Sinha, “Configuration based scene classification and image indexing”, IEEE Conf. on Computer Vision and Pattern Recognition. © 1997 IEEE.

21. Color layout
** Image adapted from Smith and Chang : Single Color Extraction and Image Query

22. Color correlograms
Pick any pixel p1 of color Ci in the image I, at distance k away from p1 pick another pixel p2, what is the probability that p2 is also of color Ci?
Red ?
Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih. Image indexing using color correlograms.  In IEEE Conference on Computer Vision and Pattern Recognition, pages , 1997. k P2 P1
Image: I

23. Color correlograms
The auto-correlogram of image I for color Ci , distance k:
Integrates both color and spatial information.
Efficient algorithms exist to compute this.

24. Color correlograms
Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih. Image indexing using color correlograms.  In IEEE Conference on Computer Vision and Pattern Recognition, pages , 1997.

25. Retrieval with texture
Can compute local texture by:
Filtering with various kernels
Taking the signature at each pixel as the filter response at the pixel for each kernel
How should texture signatures be used:
Histograms
- Image motion causes problems
- Can consider various motions separately, but more expensive
Texture segmentation
- Compare average texture signature within regions

26. Retrieval with texture
Yossi Rubner, Carlo Tomasi. Texture-Based Image Retrieval Without Segmentation. IEEE International Conference on Computer Vision, Kerkyra, Greece, September 1999, pages

27. Retrieval with texture
Yossi Rubner, Carlo Tomasi. Texture-Based Image Retrieval Without Segmentation. IEEE International Conference on Computer Vision, Kerkyra, Greece, September 1999, pages

28. Texture of textures
Positive examples (top) and retrieved results (bottom)
K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

29. Texture of textures
K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

30. Texture of textures
The best filters to use for each class are learned using boosting.
K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

31. Texture of textures

32. Retrieval using shape
We can do even better if we include shape in the query.
How do we know what shapes are in each image in the database?
Segmentation
Blobworld (UC-Berkeley) uses color, texture, shape, and location to find good matches.
Background can be disregarded, if desired.
UC Berkeley Digital Library Project.

33. Shape measures Global shape measures include: Boundary length
Area enclosed
Boundary curvature (overall or histogram)
Moments
Projections onto axes
Tangent angle histogram
However, global measures are not ideal in many situations.