0. Relevance Feedback for the Earth Mover‘s Distance
Marc Wichterich, Christian Beecks, Martin Sundermeyer, Thomas Seidl
Data Management and Data Exploration Group RWTH Aachen University, Germany

1. Introduction Distance-based Adaptable Similarity Search
Similarity of objects defined by distance function
Small distance → similar, large distance → dissimilar
Query by example: user-given object, find similar ones
Query and distance only approximate descriptions of user’s desired result
If delivered result does not meet expectations:
Bad query? Bad distance? Bad database?
How to do it better?
Relevance Feedback attempts to adapt query/similarity model based on simple user input (result relevancy)

2. RF for EMD Relevance Feedback Earth Mover’s Distance RF EMD
user DB
query, feedback
results
feedback system
similarity model
Photo: Flickr / Caro Wallis
Earth Mover’s Distance
RF for EMD RF
EMD

3. Overview Introduction Adaptive Similarity Model
Feature Signatures
The Earth Mover’s Distance
Relevance Feedback for the Earth Mover’s Distance
Experimental Evaluation
Conclusion

4. Similarity Model – Feature Signatures
color x

5. Similarity Model – Earth Mover’s Distance
Introduced in Computer Vision by Rubner et al.
Used in many differing application domains
Idea: transform features of Q into features of P
EMD: minimum of transformation cost Q P x y x y

6. Feature Transformation

7. EMD – Formal Definition
Modeled as linear optimization (transportation problem)

8. Overview Introduction Adaptive Similarity Model
Relevance Feedback for the Earth Mover’s Distance
The Feedback Loop
Query Adaptation
Heuristic EMD Adaptation
Optimization-based EMD Adaptation
Experimental Evaluation
Conclusion

9. ? The Feedback Loop no yes exit start get query retrieve results
user DB
query, feedback
results
feedback system
similarity model
start
get query
retrieve results
adapt distance ?
display results
adapt query
satisfied? no
get feedback
yes
exit

10. Query Adaptation Input: signatures from relevant objects
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Query Adaptation
Input: signatures from relevant objects
Output: new query signature
Idea: cluster signature elements
Refinements by Rubner:
Only keep clusters with elements from majority of signatures
Reweight resulting signature accordingly
Combine with fixed gd L2 and call it „Query-by-Refinement“
„Query-by-Refinement“ is baseline for our evaluation
We adapt EMD via ground distance

11. Heuristic EMD Adaptation 1
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Heuristic EMD Adaptation 1
Approach: pick gd based on feedback
gd should reflect user preferences:
Don’t care if blue cluster at upper half of image is moved left/right
Do care if it is moved vertically
Use variance information in relevant feedback
Low variance → assume user cares
High variance → assume user does not care
Measure variance in feedback locally around query signature elements ci(Q).
Define gd: c(Q) x FS → R ( )

12. Heuristic EMD Adaptation 2
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Heuristic EMD Adaptation 2
Not 1 but m distance functions:
gdi(ci(Q),y) = ((ci(Q)- y) Vi (ci(Q)- y)T)½
Weighted Euclidean Distances (weights on diagonal of Vi)
Vi : inverted variance for ci(Q) per feature space dimension

13. Optimization-Based EMD Adaptation 1
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Optimization-Based EMD Adaptation 1
Aim: Pick best possible gd.
Failback: Find a good one.
Q: When is gd good? A: If ranking it produces is good.
New Q: When is a ranking of DB good?
Given ground truth, a number of measures exist
We used “average precision at relevant positions”
We have ground truth for part of the DB: feedback
Idea: test candidates for gd on feedback
Ranking
Avg. Precision
1.000
0.854
0.365

14. Optimization-Based EMD Adaptation 2
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Optimization-Based EMD Adaptation 2
Optimization:
Optimization variable: gd
Objective function: avgPrec(EMDgd , q, Feedback)
Constraints: m weighted Euclidean distances
Analytic optimization with closed form for weights infeasible (ranking/sorting, EMDs in objective function)
Probabilistic optimization via Simulated Annealing
Start with some initial solution
Move in solution space
Compute objective function
Adopt solution with certain probability
Iterate & turn more greedy

15. Optimization-Based EMD Adaptation 3
satisfied?
retrieve results
exit
start
get query
query
distance
feedback
display results
Optimization-Based EMD Adaptation 3
Optimization for EMD based on Feedback:
Solution: weights for m weighted Euclidean distances
Initial solution: given by heuristic
Moving: redistribute weights per Euclidean distance
Objective function: avgPrec(EMDgd , q, Feedback)
Results for EMDgd on DB?

16. Overview Introduction Adaptive Similarity Model
Relevance Feedback for the Earth Mover’s Distance
Experimental Evaluation
Conclusion

17. Experimental Evaluation: Databases
72,000 images in ALOI DB
~60,000 images in COREL DB

18. Experimental Evaluation: ALOI
Query-by-Refinement
Heuristic Adaptation
Optimization-based

19. Experimental Evaluation: COREL
Query-by-Refinement
Heuristic Adaptation
Optimization-based

20. Experimental Evaluation
After 5 iterations of looking for doors in COREL:
(a) Query-by-Refinement (b) Heuristic (c) Optimization-Based
pos

21. Conclusion Exploited adaptability of the EMD in RF framework
Goal: Improve similarity search results
Techniques:
Baseline: fixed ground distance
Statistics-based heuristic adaptation
Optimization-based adaptation
Evaluation:
Experiments on two image datasets
More relevant objects in fewer iterations
Techniques extensible to other adaptable distance functions