1. Relevance Feedback Content-Based Image Retrieval Using Query Distribution Estimation Based on Maximum Entropy Principle
Irwin King and Zhong Jin
The Chinese University of Hong Kong
ICONIP2001, Shanghai, China
November 16, 2001
This paper is supported in part by an Earmarked Grant from the Hong Kong Research Grants Council #CUHK4407/99E.

2. Introduction Images Internet Audio & Video Information = power!
Management of information is important
Information retrieval is crucial
Search Engine
Images
Internet
Text
ICONIP2001, Shanghai
2001/11/16

3. Content-Based Image Retrieval
Retrieve images by
color, shape, texture, sketch, spatial organization, etc.
ICONIP2001, Shanghai
2001/11/16

4. Relevance Feedback Relevance feedback image retrieval
Using user’s response to refine the retrieval result iteratively
Why is relevance feedback significant?
It is difficult for user to describe the desired outcome using features
Only user’s similarity information is available
Example
Face identification
ICONIP2001, Shanghai
2001/11/16

5. Presentation User’s Feedback Generation Generation
ICONIP2001, Shanghai
2001/11/16

6. Key Issues Goal Two problems
To utilize user’s feedback to estimate the outcome
To construct an on-line and real time user classifier for the query to prune out unwanted data
Two problems
How to generate the images to be selected?
How to utilize user’s feedback to update the image’s relevance?
ICONIP2001, Shanghai
2001/11/16

7. Related Work Relevance Feedback
Cox et al. used a Bayesian approach to update each image’s relevance
Rui et al. used a simple neural learning-like iterative updating equation to update each image’s relevance
Problems
Calculation is over ALL images=> time consuming
Unnatural image selection procedure for presentation=> no logical basis which leads to poor performance
ICONIP2001, Shanghai
2001/11/16

8. Proposed Framework Estimation Stage Generation Stage
Estimate the feedback mean and the feedback covariance matrix by using accumulative relevant and irrelevant retrieval information.
This is a local estimate, not a global one!
Generation Stage
Generate a set of inquiries for relevance selection based on Maximum Entropy Principle.
The MEP gives a natural and logical guideline to select presentation images.
ICONIP2001, Shanghai
2001/11/16

9. An Illustration
ICONIP2001, Shanghai
2001/11/16

10. Outline of Parameters Given a set of data points
D={di}, i=1,…,n
Image generation for presentation and selection
G = {dj}, G is a subset of D
Relevance feedback from the user
RF = {dk}, RF is a subset of G
Properties
Sequential revelation of information from the user
The relevance feedback information is relatively sparse
Possible user’s misinformation
ICONIP2001, Shanghai
2001/11/16

11. Estimation Stage Assume parametric model based on normal distribution
When the number T of relevant retrieval is less than the dimension M of the feature space, it is assumed that
When T=1, an estimation can be given
by an equal-probability constrain
ICONIP2001, Shanghai
2001/11/16

12. Generation Stage
Using MEP, we want to maximize the information obtained from the user’s feedback.
For K number of retrievals, K+1 points can be determined according to the following equal-probability conditions:
all similar images in the database can be divided in the following K subsets:
where
ICONIP2001, Shanghai
2001/11/16

13. Experiments Trademark image database 1,400 128 x 128 trademark images
ICONIP2001, Shanghai
2001/11/16

14. Shape Feature Extraction
Shape content-based retrieval
7 dimensional invariant moments
ICONIP2001, Shanghai
2001/11/16

15. Test by Deformation
We vary the trademark images using 10 deformation transformations.
ICONIP2001, Shanghai
2001/11/16

16. Test Images
100 Test Images:
ICONIP2001, Shanghai
2001/11/16

17. Experimental Aim Experimental goal
To evaluate the efficiency of the proposed generation stage, Generation MAXENT.
The retrieval performance is measured using the following Average Retrieval Precision (ARP):
Where K=10.
ICONIP2001, Shanghai
2001/11/16

18. Experimental Procedure
Compare three distances
The three-step experiments are designed as follows:
For a query image, return K retrievals by Euclidean distance.
Perform the estimation stage and return retrievals by Euclidean distance, Mahalanobis distance, and Generation MAXENT respectively.
Perform the estimation stage and return K retrievals by using the Mahalanobis distance.
ICONIP2001, Shanghai
2001/11/16

19. Experimental Results http://www.cse.cuhk.edu.hk/~miplab/MAXENT
ICONIP2001, Shanghai
2001/11/16

20. Analysis of Results
The proposed generation stage Generation MAXENT outperforms the commonly used Euclidean distance and Mahalanobis distance
MAXENT begins with very large error, but it converges much quicker than the other two distance measures.
The proposed Generation MAXENT aims to retrieve image samples which can reflect the user’s query distribution function better.
ICONIP2001, Shanghai
2001/11/16

21. Discussions Theoretical convergence properties
How to ensure convergence
How to minimize the number of iterations
Better heuristics for the generation stage
How to select images for presentation efficiently
Better error handling
How to handle user’s mistake when selecting relevant images
Better experimental procedure
How to benchmark relevance feedback results
ICONIP2001, Shanghai
2001/11/16

22. Conclusion
Proposed a novel two-stage relevance feedback framework for content-based image retrieval
Query estimation through user’s feedback
Maximum Entropy Principle for image generation in the presentation stage
They are shown to be successful in improving accuracy and speed on a simple trademark image database.
ICONIP2001, Shanghai
2001/11/16

23. Introduction Content-Based Image retrieval (CBIR) :
The selection of images from a collection via primitive visual features representing color, shape, and texture extracted from images themselves.
Successful CBIR systems require the integration of various techniques in the fields of :
Pattern Recognition (PR)
Digital Image Processing (DIP)
Information Retrieval (IR)
ICONIP2001, Shanghai
2001/11/16

24. Introduction Relevance feedback (RF) : RF techniques include mainly:
An iterative and interactive process for query reformulation based on user's feedback.
RF techniques include mainly:
Query moving technique
Similarity function re-weighting technique
ICONIP2001, Shanghai
2001/11/16

25. Key Problem
Problems
Since the retrievals under the commonly used nearest-neighbor rule cannot reflect the query distribution function properly , most of relevance feedback techniques may fail under the following assumption:
The number of relevant retrievals is small
The number of iterations is required to be small
ICONIP2001, Shanghai
2001/11/16

26. Background
RF technique in CBIR system can be regarded as a form of two-stage automatic learning for the unknown query distribution function:
Estimate the query distribution function by using the Expectation-Maximization (EM) algorithm or by the classical statistics.
Generate the inquiries to be returned to the user, where the nearest-neighbor rule is commonly used.
ICONIP2001, Shanghai
2001/11/16

27. Background Limitation of estimation theories:
EM has its limitations in CBIR Because of a small number of labeled data in RF.
Only relevant information can be utilized by classical statistical theory.
ICONIP2001, Shanghai
2001/11/16

28. Background Limitation of the nearest-neighbor rule:
the retrievals generated cannot wholly reflect the Query Distribution Function (QDF) because the underlying QDF may not be isotropic in nature.
QDF is the statistical distribution function deformed by all the images similar to the given query image in high dimensional feature space.
ICONIP2001, Shanghai
2001/11/16

29. Background Review Shannon’s Entropy Maximum Entropy Principle (MEP):
To obtain estimations by determining a probability distribution associated with a random variable over a discrete space which has the greatest entropy subject to constraints on the expectations of a given set of functions of the variable.
The Maximum Entropy (MAXENT) solution with no bias (or constraints) is
ICONIP2001, Shanghai
2001/11/16

30. Background Review Some work on IR by MEP:
In the early 80's, Cooper et al. made a strong case for applying the maximum entropy approach to the problems of information retrieval.
Kantor extended the analysis of the MEP in the context of information retrieval.
Recently, Greiff and Ponte took a fresh look at modeling approaches to information retrieval and analyzed classical probabilistic IR models in light of the MEP.
ICONIP2001, Shanghai
2001/11/16