Presentation is loading. Please wait.

Presentation is loading. Please wait.

Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University

Published byAileen Fields Modified over 9 years ago

Similar presentations

Presentation on theme: "Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University"— Presentation transcript:

1 Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr

2 CS 484, Spring 2015©2015, Selim Aksoy2 Image retrieval Searching a large database for images that match a query: What kind of databases? What kind of queries? What constitutes a match? How do we make such searches efficient?

3 CS 484, Spring 2015©2015, Selim Aksoy3 Applications Art Collections Fine Arts Museum of San Francisco Medical Image Databases CT, MRI, Ultrasound, The Visible Human Scientific Databases Earth Sciences General Image Collections for Licensing Corbis, Getty Images The World Wide Web Google, Microsoft, Flickr

4 CS 484, Spring 2015©2015, Selim Aksoy4 Corel data set 60,000 images with annotated keywords

5 CS 484, Spring 2015©2015, Selim Aksoy5 Fine Arts Museum of San Francisco 80,000 images

6 CS 484, Spring 2015©2015, Selim Aksoy6 Query formulation Text description (keywords) Query by example Query by sketch Symbolic description (man and woman on a beach) Relevance feedback

7 CS 484, Spring 2015©2015, Selim Aksoy7 Google query on “rose”

8 CS 484, Spring 2015©2015, Selim Aksoy8 Corel query on “rose”

9 CS 484, Spring 2015©2015, Selim Aksoy9 Corbis query on “rose”

10 CS 484, Spring 2015©2015, Selim Aksoy10 Difficulties with keywords Images may not have keywords. (An image is worth … how many key-words?) Query is not easily satisfied by keywords. “A casually dressed couple gazing into each others eyes lovingly with dramatic clouds in the background.” “Pretty girl doing something active, sporty in a summery setting, beach - not wearing lycra, exercise clothes - more relaxed in tee-shirt. Feature is about deodorant so girl should look active - not sweaty but happy, healthy, carefree - nothing too posed or set up - nice and natural looking.”  Content-based image retrieval (CBIR)

11 CS 484, Spring 2015©2015, Selim Aksoy11 Content-based image retrieval Image Database Query Image Distance Measure Retrieved Images Image Feature Extraction User Feature Space Images

12 CS 484, Spring 2015©2015, Selim Aksoy12 Image representations and features Image representations: Iconic Global Region-based Object-based Image features: Color Texture Shape Objects and their relationships (this is the most powerful, but you have to be able to recognize the objects!)

13 CS 484, Spring 2015©2015, Selim Aksoy13 Image similarity Distance measures: Euclidean distance Other L p metrics Histogram intersection Cosine distance Earth mover’s distance Probabilistic similarity measures: P( relevance | two images ) P( relevance | two images ) / P( irrelevance | two images)

14 CS 484, Spring 2015©2015, Selim Aksoy14 Global histograms Searching using global color histograms

15 CS 484, Spring 2015©2015, Selim Aksoy15 Global histograms “Airplanes” using color histograms (4/12)“Sunsets” using Gabor texture (3/12)

16 CS 484, Spring 2015©2015, Selim Aksoy16 Query by image content (QBIC)

17 CS 484, Spring 2015©2015, Selim Aksoy17 Color histograms in QBIC The QBIC color histogram distance is: d hist (I,Q) = (h(I) - h(Q)) T A (h(I) - h(Q)). h(I) is a K-bin histogram of a database image. h(Q) is a K-bin histogram of the query image. A is a K x K similarity matrix. How similar is blue to cyan? R G B Y C V 1 0 0.5 0.5 0 1 0.5.5 0 0 0 1 1 RGBYCVRGBYCV ? ?

18 CS 484, Spring 2015©2015, Selim Aksoy18 Color percentages in QBIC %40 red, %30 yellow, %10 black

19 CS 484, Spring 2015©2015, Selim Aksoy19 Color layout in QBIC

20 CS 484, Spring 2015©2015, Selim Aksoy20 Earth mover’s distance

21 CS 484, Spring 2015©2015, Selim Aksoy21 Earth mover’s distance Visualization using EMD and multidimensional scaling

22 CS 484, Spring 2015©2015, Selim Aksoy22 Probabilistic similarity measures Two classes: Relevance class A Irrelevance class B Bayes classifier Assign (  i,  j ) to Discriminant function for classification Rank images according to posterior ratio values based on feature differences.

23 CS 484, Spring 2015©2015, Selim Aksoy23 Probabilistic similarity measures “Residential interiors” (12/12)“Fields” (12/12)

24 CS 484, Spring 2015©2015, Selim Aksoy24 Shape-based retrieval

25 CS 484, Spring 2015©2015, Selim Aksoy25 Shape-based retrieval

26 CS 484, Spring 2015©2015, Selim Aksoy26 Elastic shape matching

27 CS 484, Spring 2015©2015, Selim Aksoy27 Iconic matching

28 CS 484, Spring 2015©2015, Selim Aksoy28 Iconic matching Wavelet-based image compression Quantization of coefficients

29 CS 484, Spring 2015©2015, Selim Aksoy29 Iconic matching

30 CS 484, Spring 2015©2015, Selim Aksoy30 Region-based retrieval: Blobworld

31 CS 484, Spring 2015©2015, Selim Aksoy31 Region-based retrieval: Blobworld

32 CS 484, Spring 2015©2015, Selim Aksoy32 Retrieval using spatial relationships Build graph using regions and their spatial relationships. Similarity is computed using graph matching. sky sand tiger grass above adjacent above inside above adjacent image abstract regions

33 CS 484, Spring 2015©2015, Selim Aksoy33 Combining multiple features Text query on “rose”

34 CS 484, Spring 2015©2015, Selim Aksoy34 Combining multiple features Visual query on

35 CS 484, Spring 2015©2015, Selim Aksoy35 Combining multiple features Text query on “rose” and visual query on

36 CS 484, Spring 2015©2015, Selim Aksoy36 Video Google: object matching

37 CS 484, Spring 2015©2015, Selim Aksoy37 Video Google Viewpoint invariant descriptors Visual vocabulary

38 CS 484, Spring 2015©2015, Selim Aksoy38 Video Google Now is the time for all good men to come to the aid of their country. Summer has come and passed. The innocent can never last. Document 1 Document 2 WordDocument aid1 all1 and2 can2 come1, 2 country1 for1 good1 …… the1, 1, 2 …… Inverted index

39 CS 484, Spring 2015©2015, Selim Aksoy39 Video skimming

40 CS 484, Spring 2015©2015, Selim Aksoy40 Event detection, indexing, retrieval

41 CS 484, Spring 2015©2015, Selim Aksoy41 Informedia Digital Video Library IDVL interface returned for "El Nino" query along with different multimedia abstractions from certain documents.

42 CS 484, Spring 2015©2015, Selim Aksoy42 Informedia Digital Video Library IDVL interface returned for “bin ladin" query. The results can be tuned using many classifiers.

43 CS 484, Spring 2015©2015, Selim Aksoy43 Relevance feedback In real interactive CBIR systems, the user should be allowed to interact with the system to “refine” the results of a query until he/she is satisfied.

44 CS 484, Spring 2015©2015, Selim Aksoy44 Relevance feedback Example methods: Query point movement Query point is moved toward positive examples and moved away from negative examples. Weighting features The CBIR system should automatically adjust the weight that were given by the user for the relevance of previously retrieved documents. Weighting similarity measures Feature density estimation Probabilistic relevance feedback

45 CS 484, Spring 2015©2015, Selim Aksoy45 Relevance feedback Positive feedback Negative feedback

46 CS 484, Spring 2015©2015, Selim Aksoy46 Relevance feedback “Sunsets” using color histograms (1/12) Using combined features (6/12)After 1 st feedback (12/12)

47 CS 484, Spring 2015©2015, Selim Aksoy47 Relevance feedback “Auto racing” using color histograms (3/12) Using combined features (9/12)After 1 st feedback (12/12)

48 CS 484, Spring 2015©2015, Selim Aksoy48 Indexing for fast retrieval Use of key images and the triangle inequality for efficient retrieval.

49 CS 484, Spring 2015©2015, Selim Aksoy49 Indexing for fast retrieval Offline 1. Choose a small set of key images. 2. Store distances from database images to keys. Online (given query Q) 1. Compute the distance from Q to each key. 2. Obtain lower bounds on distances to database images. 3. Threshold or return all images in order of lower bounds.

50 CS 484, Spring 2015©2015, Selim Aksoy50 Indexing for fast retrieval Hierarchical cellular tree

51 CS 484, Spring 2015©2015, Selim Aksoy51 Indexing for fast retrieval

52 CS 484, Spring 2015©2015, Selim Aksoy52 Performance evaluation Two traditional measures for retrieval performance in the information retrieval literature are precision and recall. Given a particular number of images retrieved, precision is defined as the percentage of retrieved images that are actually relevant, and recall is defined as the percentage of relevant images that are retrieved.

53 CS 484, Spring 2015©2015, Selim Aksoy53 Current research objective Image Database Query ImageRetrieved Images Object-oriented Feature Extraction User Images boat …  Animals  Buildings  Office Buildings  Houses  Transportation Boats Vehicles … Categories

54 CS 484, Spring 2015©2015, Selim Aksoy54 Demos Google Image Search (http://images.google.com/)http://images.google.com/ Microsoft Image Search (https://www.bing.com/images)https://www.bing.com/images Yahoo Image Search (http://images.search.yahoo.com/)http://images.search.yahoo.com/ Vitalas ( http://vitalas.ercim.eu/) http://vitalas.ercim.eu/ FIDS (http://www.cs.washington.edu/research/ imagedatabase/demo/fids/)http://www.cs.washington.edu/research/ imagedatabase/demo/fids/ Like Visual Shopping (http://www.like.com/)http://www.like.com/ Now Google shopping (http://www.google.com/shopping)http://www.google.com/shopping Google Similar Images http://googleblog.blogspot.com/2009/10/similar-images-graduates-from- google.html http://googleblog.blogspot.com/2009/10/similar-images-graduates-from- google.html Google Image Swirl http://googleresearch.blogspot.com/2009/11/explore-images-with-google- image-swirl.html http://googleresearch.blogspot.com/2009/11/explore-images-with-google- image-swirl.html

Download ppt "Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University"

Similar presentations

Image Retrieval With Relevant Feedback Hayati Cam & Ozge Cavus IMAGE RETRIEVAL WITH RELEVANCE FEEDBACK Hayati CAM Ozge CAVUS.

Image Retrieval With Relevant Feedback Hayati Cam & Ozge Cavus IMAGE RETRIEVAL WITH RELEVANCE FEEDBACK Hayati CAM Ozge CAVUS.
Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki 2008-11629.

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University

Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University
Content-based Image Retrieval (CBIR) Searching a large database for images that match a query: –What kinds of databases? –What kinds of queries? –What.

Content-based Image Retrieval (CBIR) Searching a large database for images that match a query: –What kinds of databases? –What kinds of queries? –What.
Relevance Feedback and User Interaction for CBIR Hai Le Supervisor: Dr. Sid Ray.

Relevance Feedback and User Interaction for CBIR Hai Le Supervisor: Dr. Sid Ray.
Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University

Content-Based Image Retrieval Selim Aksoy Department of Computer Engineering Bilkent University
Introduction to Information Retrieval

Introduction to Information Retrieval
Multimedia Database Systems

Multimedia Database Systems
Three things everyone should know to improve object retrieval

Three things everyone should know to improve object retrieval
Discriminative Relevance Feedback With Virtual Textual Representation For Efficient Image Retrieval Suman Karthik and C.V.Jawahar.

Discriminative Relevance Feedback With Virtual Textual Representation For Efficient Image Retrieval Suman Karthik and C.V.Jawahar.
Image Retrieval Basics Uichin Lee KAIST KSE Slides based on “Relevance Models for Automatic Image and Video Annotation & Retrieval” by R. Manmatha (UMASS)

Image Retrieval Basics Uichin Lee KAIST KSE Slides based on “Relevance Models for Automatic Image and Video Annotation & Retrieval” by R. Manmatha (UMASS)
Image Information Retrieval Shaw-Ming Yang IST 497E 12/05/02.

Image Information Retrieval Shaw-Ming Yang IST 497E 12/05/02.
Chapter 8 Content-Based Image Retrieval. Query By Keyword: Some textual attributes (keywords) should be maintained for each image. The image can be indexed.

Chapter 8 Content-Based Image Retrieval. Query By Keyword: Some textual attributes (keywords) should be maintained for each image. The image can be indexed.
1 Overview of Image Retrieval Hui-Ying Wang. 2/42 Reference Smeulders, A. W., Worring, M., Santini, S., Gupta, A.,, and Jain, R. 2000. “Content-based.

1 Overview of Image Retrieval Hui-Ying Wang. 2/42 Reference Smeulders, A. W., Worring, M., Santini, S., Gupta, A.,, and Jain, R “Content-based.
ARNOLD SMEULDERS MARCEL WORRING SIMONE SANTINI AMARNATH GUPTA RAMESH JAIN PRESENTERS FATIH CAKIR MELIHCAN TURK Content-Based Image Retrieval at the End.

ARNOLD SMEULDERS MARCEL WORRING SIMONE SANTINI AMARNATH GUPTA RAMESH JAIN PRESENTERS FATIH CAKIR MELIHCAN TURK Content-Based Image Retrieval at the End.
Lecture 12 Content-Based Image Retrieval

Lecture 12 Content-Based Image Retrieval
1 Content-Based Retrieval (CBR) -in multimedia systems Presented by: Chao Cai Date: March 28, 2006 C SC 561.

1 Content-Based Retrieval (CBR) -in multimedia systems Presented by: Chao Cai Date: March 28, 2006 C SC 561.
Content Based Image Clustering and Image Retrieval Using Multiple Instance Learning Using Multiple Instance Learning Xin Chen Advisor: Chengcui Zhang Department.

Content Based Image Clustering and Image Retrieval Using Multiple Instance Learning Using Multiple Instance Learning Xin Chen Advisor: Chengcui Zhang Department.
ISP 433/533 Week 2 IR Models.

ISP 433/533 Week 2 IR Models.
Image Search Presented by: Samantha Mahindrakar Diti Gandhi.

Image Search Presented by: Samantha Mahindrakar Diti Gandhi.

Similar presentations

Ads by Google