Image Retrieval: Current Techniques, Promising Directions, and Open Issues Yong Rui, Thomas Huang and Shih-Fu Chang Published in the Journal of Visual Communication and Image Representation. Presented by: Deepak Bote

Presentation Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

History of Image Retrieval Traditional text-based image search engines Traditional text-based image search engines Manual annotation of images Manual annotation of images Use text-based retrieval methods Use text-based retrieval methods E.g. E.g. Water lilies Flowers in a pond

Limitations of text-based approach Problem of image annotation Problem of image annotation Large volumes of databases Large volumes of databases Valid only for one language – with image retrieval this limitation should not exist Valid only for one language – with image retrieval this limitation should not exist Problem of human perception Problem of human perception Subjectivity of human perception Subjectivity of human perception Too much responsibility on the end-user Too much responsibility on the end-user Problem of deeper (abstract) needs Problem of deeper (abstract) needs Queries that cannot be described at all, but tap into the visual features of images. Queries that cannot be described at all, but tap into the visual features of images.

Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

What is CBIR? Images have rich content. Images have rich content. This content can be extracted as various content features: This content can be extracted as various content features: Mean color, Color Histogram etc… Mean color, Color Histogram etc… Take the responsibility of forming the query away from the user. Take the responsibility of forming the query away from the user. Each image will now be described by its own features. Each image will now be described by its own features.

CBIR – A sample search query User wants to search for, say, many rose images User wants to search for, say, many rose images He submits an existing rose picture as query. He submits an existing rose picture as query. He submits his own sketch of rose as query. He submits his own sketch of rose as query. The system will extract image features for this query. The system will extract image features for this query. It will compare these features with that of other images in a database. It will compare these features with that of other images in a database. Relevant results will be displayed to the user. Relevant results will be displayed to the user.

Sample Query

Sample CBIR architecture

Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

Feature Extraction What are image features? What are image features? Primitive features Primitive features Mean color (RGB) Mean color (RGB) Color Histogram Color Histogram Semantic features Semantic features Color Layout, texture etc… Color Layout, texture etc… Domain specific features Domain specific features Face recognition, fingerprint matching etc… Face recognition, fingerprint matching etc… General features

Mean Color Pixel Color Information: R, G, B Pixel Color Information: R, G, B Mean component (R,G or B)= Mean component (R,G or B)= Sum of that component for all pixels Sum of that component for all pixels Number of pixels Pixel

Histogram Frequency count of each individual color Frequency count of each individual color Most commonly used color feature representation Most commonly used color feature representation Image Corresponding histogram

Color Layout Need for Color Layout Need for Color Layout Global color features give too many false positives Global color features give too many false positives How it works: How it works: Divide whole image into sub-blocks Divide whole image into sub-blocks Extract features from each sub-block Extract features from each sub-block Can we go one step further? Can we go one step further? Divide into regions based on color feature concentration Divide into regions based on color feature concentration This process is called segmentation. This process is called segmentation.

Example: Color layout ** Image adapted from Smith and Chang : Single Color Extraction and Image Query

Texture Texture – innate property of all surfaces Texture – innate property of all surfaces Clouds, trees, bricks, hair etc… Clouds, trees, bricks, hair etc… Refers to visual patterns of homogeneity Refers to visual patterns of homogeneity Does not result from presence of single color Does not result from presence of single color Most accepted classification of textures based on psychology studies – Tamura representation Most accepted classification of textures based on psychology studies – Tamura representation Coarseness Contrast Directionality Linelikeness Regularity Roughness

Segmentation issues Considered as a difficult problem Considered as a difficult problem Not reliable Not reliable Segments regions, but not objects Segments regions, but not objects Different requirements from segmentation: Different requirements from segmentation: Shape extraction: High Accuracy required Shape extraction: High Accuracy required Layout features: Coarse segmentation may be enough Layout features: Coarse segmentation may be enough

Presentation Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

Problem of high dimensions Mean Color = RGB = 3 dimensional vector Mean Color = RGB = 3 dimensional vector Color Histogram = 256 dimensions Color Histogram = 256 dimensions Effective storage and speedy retrieval needed Effective storage and speedy retrieval needed Traditional data-structures not sufficient Traditional data-structures not sufficient R-trees, SR-Trees etc… R-trees, SR-Trees etc…

2-dimensional space D1 D2 Point A

3-dimensional space

Now, imagine… An N-dimensional box!! An N-dimensional box!! We want to conduct a nearest neighbor query. We want to conduct a nearest neighbor query. R-trees are designed for speedy retrieval of results for such purposes R-trees are designed for speedy retrieval of results for such purposes Designed by Guttmann in 1984 Designed by Guttmann in 1984

Presentation Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

IBM’s QBIC QBIC – Query by Image Content QBIC – Query by Image Content First commercial CBIR system. First commercial CBIR system. Model system – influenced many others. Model system – influenced many others. Uses color, texture, shape features Uses color, texture, shape features Text-based search can also be combined. Text-based search can also be combined. Uses R*-trees for indexing Uses R*-trees for indexing

QBIC – Search by color ** Images courtesy : Yong Rao

QBIC – Search by shape ** Images courtesy : Yong Rao

QBIC – Query by sketch ** Images courtesy : Yong Rao

Virage Developed by Virage inc. Developed by Virage inc. Like QBIC, supports queries based on color, layout, texture Like QBIC, supports queries based on color, layout, texture Supports arbitrary combinations of these features with weights attached to each Supports arbitrary combinations of these features with weights attached to each This gives users more control over the search process This gives users more control over the search process

VisualSEEk Research prototype – University of Columbia Research prototype – University of Columbia Mainly different because it considers spatial relationships between objects. Mainly different because it considers spatial relationships between objects. Global features like mean color, color histogram can give many false positives Global features like mean color, color histogram can give many false positives Matching spatial relationships between objects and visual features together result in a powerful search. Matching spatial relationships between objects and visual features together result in a powerful search.

ISearch – my own system

Feature selection in ISearch

Database Admin facility in ISearch

Presentation Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

Open issues Gap between low level features and high-level concepts Gap between low level features and high-level concepts Human in the loop – interactive systems Human in the loop – interactive systems Retrieval speed – most research prototypes can handle only a few thousand images. Retrieval speed – most research prototypes can handle only a few thousand images. A reliable test-bed and measurement criterion, please! A reliable test-bed and measurement criterion, please!

Presentation Outline History of image retrieval – Issues faced History of image retrieval – Issues faced Solution – Content-based image retrieval Solution – Content-based image retrieval Feature extraction Feature extraction Multidimensional indexing Multidimensional indexing Current Systems Current Systems Open issues Open issues Conclusion Conclusion

Conclusion Satisfactory progress, but still… Satisfactory progress, but still… A long way to go…!!

Acknowledgements Dr. Padma Mundur Dr. Padma Mundur Mr. Yong Rao Mr. Yong Rao Mr. Sumit Jain, Software Engineer, KPIT Cummins, India Mr. Sumit Jain, Software Engineer, KPIT Cummins, India Mr. Ajay Joglekar, Software Engineer, Veritas India. Mr. Ajay Joglekar, Software Engineer, Veritas India.

References Y. Rui, T. S. Huang, and S.-F. Chang, “Image retrieval: Current techniques, promising directions, and open issues” Y. Rui, T. S. Huang, and S.-F. Chang, “Image retrieval: Current techniques, promising directions, and open issues” S. Jain, A. Joglekar, and D. Bote, ISearch: A Content-based Image Retrieval (CBIR) Engine, as Bachelor of Computer Engineering final year thesis, Pune University, 2002 S. Jain, A. Joglekar, and D. Bote, ISearch: A Content-based Image Retrieval (CBIR) Engine, as Bachelor of Computer Engineering final year thesis, Pune University, 2002

THANK YOU!!! THANK YOU!!! Questions? Questions?