Lecture 12 Content-Based Image Retrieval Slides by: Deepak Bote, Xiaoguang Feng, David A. Forsyth, Clark F. Olson, Yossi Rubner, Linda G. Shapiro

Image databases Large collections of image (and video) occur in many applications: Stock photos and footage Military World-wide web Medical imaging Space exploration Surveillance Many others…

Image databases Most collections of images and videos are not image databases. No DBMS manages the data No facility for complex queries is available This is a rapidly expanding area of interest in computer vision: How can we find a particular image that we are interested in? Can we locate images that meet some description? How can we organize the collection meaningfully? Can we extract new information by exploring the collection automatically?

History

Image retrieval by annotation Traditional text-based image search engines Manual annotation of images Use text-based retrieval methods E.g. Water lilies Flowers in a pond <Its biological name>

Limitations of annotation Problem of image annotation Large volumes of databases Valid only for one language – with image retrieval this limitation should not exist Problem of human perception Subjectivity of human perception Too much responsibility on the end-user Problem of deeper (abstract) needs Queries that cannot be described at all, but tap into the visual features of images

Image retrieval Image have rich content that can be used for retrieval! The problem that has been examined the most is: How can I find images that meet some description or are like an example? This is often called “content-based image retrieval” (CBIR). Image content is used, rather than metadata. Using example images: Takes the responsibility of forming the query away from the user. Allows each image to be described by its own features. Mean color, histogram, etc.

Measures No system is perfect. Usually measured using two criteria: Recall: percentage of correct items found Precision: percentage of items found that are correct Which is more important?

User interaction Due to the imperfections in current methods, most systems have a user “in the loop”. Retrieve-refine-retrieve cycle Y. Rubner, C. Tomasi, L.J. Guibas, “A metric for distributions with applications to image databases”, Intl. Conf. Computer Vision. © 1998, IEEE.

Features for retrieval What features are used for retrieving images? Color - Mean - Overall distribution - Relative locations Texture - Linear filters - Textures of textures (!?) Shape - Sketches - Segmented objects Others

Color histograms An early (and still popular) similarity measurement uses color histograms. The RGB (or another) color space is discretized into bins. For each bin, a count is maintained on the number of pixels that fall into the bin (since they have the right color) Once constructed, the histograms can be compared using several metrics. UC Berkeley Digital Library Project. http://elib.cs.berkeley.edu/

Color histograms The QBIC system (IBM) was the first commercial system created. It uses color, texture, shape, location, and keywords. h(I) is a K-bin histogram of a database image h(Q) is a K-bin histogram of the query image A is a K x K similarity matrix The QBIC color histogram distance is: dhist(I,Q) = (h(I) - h(Q)) A (h(I) - h(Q)) T

** Images courtesy : Yong Rao QBIC – Search by color ** Images courtesy : Yong Rao

** Images courtesy : Yong Rao QBIC – Search by shape ** Images courtesy : Yong Rao

** Images courtesy : Yong Rao QBIC – Query by sketch ** Images courtesy : Yong Rao

Histogram distances

Histogram distances

Earth mover’s distance

Earth mover’s distance

Layout templates Color histograms have no information about pixel locations. This yields false positives. We can add spatial information by using a template of the image. Example: Blue at top (sky), white below (snowy mountains), dark at bottom (mountains or lake) Simplest implementation would divide image into regular blocks. P. Lipson, E. Grimson, P, Sinha, “Configuration based scene classification and image indexing”, IEEE Conf. on Computer Vision and Pattern Recognition. © 1997 IEEE.

Color layout ** Image adapted from Smith and Chang : Single Color Extraction and Image Query

Color correlograms Pick any pixel p1 of color Ci in the image I, at distance k away from p1 pick another pixel p2, what is the probability that p2 is also of color Ci? Red ? Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih. Image indexing using color correlograms.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 762--768, 1997. k P2 P1 Image: I

Color correlograms The auto-correlogram of image I for color Ci , distance k: Integrates both color and spatial information. Efficient algorithms exist to compute this.

Color correlograms Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, and Ramin Zabih. Image indexing using color correlograms.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 762--768, 1997.

Retrieval with texture Can compute local texture by: Filtering with various kernels Taking the signature at each pixel as the filter response at the pixel for each kernel How should texture signatures be used: Histograms - Image motion causes problems - Can consider various motions separately, but more expensive Texture segmentation - Compare average texture signature within regions

Retrieval with texture Yossi Rubner, Carlo Tomasi. Texture-Based Image Retrieval Without Segmentation. IEEE International Conference on Computer Vision, Kerkyra, Greece, September 1999, pages 1018-1024.

Retrieval with texture Yossi Rubner, Carlo Tomasi. Texture-Based Image Retrieval Without Segmentation. IEEE International Conference on Computer Vision, Kerkyra, Greece, September 1999, pages 1018-1024.

Texture of textures Positive examples (top) and retrieved results (bottom) K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

Texture of textures K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

Texture of textures The best filters to use for each class are learned using boosting. K. Tieu, P. Viola, “Boosting image retrieval”, IEEE Conf. on Computer Vision and Pattern Recognition. © 2000 IEEE.

Texture of textures

Retrieval using shape We can do even better if we include shape in the query. How do we know what shapes are in each image in the database? Segmentation Blobworld (UC-Berkeley) uses color, texture, shape, and location to find good matches. Background can be disregarded, if desired. UC Berkeley Digital Library Project. http://elib.cs.berkeley.edu/

Shape measures Global shape measures include: Boundary length Area enclosed Boundary curvature (overall or histogram) Moments Projections onto axes Tangent angle histogram However, global measures are not ideal in many situations.