Relevance Feedback Content-Based Image Retrieval Using Query Distribution Estimation Based on Maximum Entropy Principle Irwin King and Zhong Jin The Chinese University of Hong Kong ICONIP2001, Shanghai, China November 16, 2001 This paper is supported in part by an Earmarked Grant from the Hong Kong Research Grants Council #CUHK4407/99E.

Introduction Images Internet Audio & Video Information = power! Management of information is important Information retrieval is crucial Search Engine Images Internet Text ICONIP2001, Shanghai 2001/11/16

Content-Based Image Retrieval Retrieve images by color, shape, texture, sketch, spatial organization, etc. ICONIP2001, Shanghai 2001/11/16

Relevance Feedback Relevance feedback image retrieval Using user’s response to refine the retrieval result iteratively Why is relevance feedback significant? It is difficult for user to describe the desired outcome using features Only user’s similarity information is available Example Face identification ICONIP2001, Shanghai 2001/11/16

Presentation User’s Feedback Generation Generation ICONIP2001, Shanghai 2001/11/16

Key Issues Goal Two problems To utilize user’s feedback to estimate the outcome To construct an on-line and real time user classifier for the query to prune out unwanted data Two problems How to generate the images to be selected? How to utilize user’s feedback to update the image’s relevance? ICONIP2001, Shanghai 2001/11/16

Related Work Relevance Feedback Cox et al. used a Bayesian approach to update each image’s relevance Rui et al. used a simple neural learning-like iterative updating equation to update each image’s relevance Problems Calculation is over ALL images=> time consuming Unnatural image selection procedure for presentation=> no logical basis which leads to poor performance ICONIP2001, Shanghai 2001/11/16

Proposed Framework Estimation Stage Generation Stage Estimate the feedback mean and the feedback covariance matrix by using accumulative relevant and irrelevant retrieval information. This is a local estimate, not a global one! Generation Stage Generate a set of inquiries for relevance selection based on Maximum Entropy Principle. The MEP gives a natural and logical guideline to select presentation images. ICONIP2001, Shanghai 2001/11/16

An Illustration ICONIP2001, Shanghai 2001/11/16

Outline of Parameters Given a set of data points D={di}, i=1,…,n Image generation for presentation and selection G = {dj}, G is a subset of D Relevance feedback from the user RF = {dk}, RF is a subset of G Properties Sequential revelation of information from the user The relevance feedback information is relatively sparse Possible user’s misinformation ICONIP2001, Shanghai 2001/11/16

Estimation Stage Assume parametric model based on normal distribution When the number T of relevant retrieval is less than the dimension M of the feature space, it is assumed that When T=1, an estimation can be given by an equal-probability constrain ICONIP2001, Shanghai 2001/11/16

Generation Stage Using MEP, we want to maximize the information obtained from the user’s feedback. For K number of retrievals, K+1 points can be determined according to the following equal-probability conditions: all similar images in the database can be divided in the following K subsets: where ICONIP2001, Shanghai 2001/11/16

Experiments Trademark image database 1,400 128 x 128 trademark images ICONIP2001, Shanghai 2001/11/16

Shape Feature Extraction Shape content-based retrieval 7 dimensional invariant moments ICONIP2001, Shanghai 2001/11/16

Test by Deformation We vary the trademark images using 10 deformation transformations. ICONIP2001, Shanghai 2001/11/16

Test Images 100 Test Images: ICONIP2001, Shanghai 2001/11/16

Experimental Aim Experimental goal To evaluate the efficiency of the proposed generation stage, Generation MAXENT. The retrieval performance is measured using the following Average Retrieval Precision (ARP): Where K=10. ICONIP2001, Shanghai 2001/11/16

Experimental Procedure Compare three distances The three-step experiments are designed as follows: For a query image, return K retrievals by Euclidean distance. Perform the estimation stage and return retrievals by Euclidean distance, Mahalanobis distance, and Generation MAXENT respectively. Perform the estimation stage and return K retrievals by using the Mahalanobis distance. ICONIP2001, Shanghai 2001/11/16

Experimental Results http://www.cse.cuhk.edu.hk/~miplab/MAXENT ICONIP2001, Shanghai 2001/11/16

Analysis of Results The proposed generation stage Generation MAXENT outperforms the commonly used Euclidean distance and Mahalanobis distance MAXENT begins with very large error, but it converges much quicker than the other two distance measures. The proposed Generation MAXENT aims to retrieve image samples which can reflect the user’s query distribution function better. ICONIP2001, Shanghai 2001/11/16

Discussions Theoretical convergence properties How to ensure convergence How to minimize the number of iterations Better heuristics for the generation stage How to select images for presentation efficiently Better error handling How to handle user’s mistake when selecting relevant images Better experimental procedure How to benchmark relevance feedback results ICONIP2001, Shanghai 2001/11/16

Conclusion Proposed a novel two-stage relevance feedback framework for content-based image retrieval Query estimation through user’s feedback Maximum Entropy Principle for image generation in the presentation stage They are shown to be successful in improving accuracy and speed on a simple trademark image database. ICONIP2001, Shanghai 2001/11/16

Introduction Content-Based Image retrieval (CBIR) : The selection of images from a collection via primitive visual features representing color, shape, and texture extracted from images themselves. Successful CBIR systems require the integration of various techniques in the fields of : Pattern Recognition (PR) Digital Image Processing (DIP) Information Retrieval (IR) ICONIP2001, Shanghai 2001/11/16

Introduction Relevance feedback (RF) : RF techniques include mainly: An iterative and interactive process for query reformulation based on user's feedback. RF techniques include mainly: Query moving technique Similarity function re-weighting technique ICONIP2001, Shanghai 2001/11/16

Key Problem Problems Since the retrievals under the commonly used nearest-neighbor rule cannot reflect the query distribution function properly , most of relevance feedback techniques may fail under the following assumption: The number of relevant retrievals is small The number of iterations is required to be small ICONIP2001, Shanghai 2001/11/16

Background RF technique in CBIR system can be regarded as a form of two-stage automatic learning for the unknown query distribution function: Estimate the query distribution function by using the Expectation-Maximization (EM) algorithm or by the classical statistics. Generate the inquiries to be returned to the user, where the nearest-neighbor rule is commonly used. ICONIP2001, Shanghai 2001/11/16

Background Limitation of estimation theories: EM has its limitations in CBIR Because of a small number of labeled data in RF. Only relevant information can be utilized by classical statistical theory. ICONIP2001, Shanghai 2001/11/16

Background Limitation of the nearest-neighbor rule: the retrievals generated cannot wholly reflect the Query Distribution Function (QDF) because the underlying QDF may not be isotropic in nature. QDF is the statistical distribution function deformed by all the images similar to the given query image in high dimensional feature space. ICONIP2001, Shanghai 2001/11/16

Background Review Shannon’s Entropy Maximum Entropy Principle (MEP): To obtain estimations by determining a probability distribution associated with a random variable over a discrete space which has the greatest entropy subject to constraints on the expectations of a given set of functions of the variable. The Maximum Entropy (MAXENT) solution with no bias (or constraints) is ICONIP2001, Shanghai 2001/11/16

Background Review Some work on IR by MEP: In the early 80's, Cooper et al. made a strong case for applying the maximum entropy approach to the problems of information retrieval. Kantor extended the analysis of the MEP in the context of information retrieval. Recently, Greiff and Ponte took a fresh look at modeling approaches to information retrieval and analyzed classical probabilistic IR models in light of the MEP. ICONIP2001, Shanghai 2001/11/16