Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 I9 CHAIR OF COMPUTER SCIENCE 9 DATA MANAGEMENT AND EXPLORATION Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support DBRank 08, April 12 th 2008, Cancún, Mexico Marc Wichterich, Christian Beecks, Thomas Seidl

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Outline Motivation Ranking DB according to Earth Movers Distance Search for suitable ground distance via user interaction Relevance Feedback The MindReader approach Challenges in multimedia context History – Change of user preferences over time Foresight – Fast exploration Conclusion and Outlook 1

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Transform object features to match those of other object Minimum work for transformation: EMD [1] Feature signatures: {(center 1, weight 1 ), (c 2,w 2 ), …} signature of object 1 signature of object 2 EMD weight assignment Motivation: Ranking according to Earth Movers Distance 2 [1] Rubner, Tomasi, Guibas, A metric for distributions with applications to image databases, in IEEE ICCV 1998.

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Requires ground distance gd in feature space gd(blue/left, purple/right) vs. gd(blue/left, red/middle) ? gd? gd? Possibly complex gd: Blue may move horizontally at low cost if at top of image (sky) Idea: Find gd according to user preferences Motivation: Ranking according to Earth Movers Distance 3

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Collecting preference information on feature space Utilize histogram-based Relevance Feedback system Histogram dimensions correspond to points in feature space System has to deliver information on histogram dimension pairs Define gd on feature space Rank DB according to EMD gd on signatures Motivation: Ranking according to Earth Movers Distance 4 feature space histogram

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 1.MR shows candidate objects 2.User rates relevant objects 3.MindReader determines: new query point q similarity matrix S for ellipsoid-shaped distance 4.Goto 1 Similarity matrix S is (pseudo) inverse covariance matrix S reflects user preferences w.r.t. histograms dimensions Relevance Feedback: MindReader Approach [2] 5 [2] Ishikawa, Subramanya, Faloutsos, MindReader: Querying databases through multiple examples, VLDB 1998.

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 MindReader: Challenges in multimedia context 6 [3]Ye, Xu, Similarity measure learning for image retrieval using feature subspace analysis, ICCIMA 2003.

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 No information only true within single iteration Idea: save information from previous rounds + = iteration k-1 iteration k result Technique: Incrementally compute weighted covariance matrix Exponential aging for ratings of previous iterations Include relevant points from all previous iterations Relevance Feedback with History (1) 7

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Relevance Feedback with History (2) 8 = 0.1 = 0.3

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Relevance Feedback with History (Summary) Feedback information crosses iteration boundaries Parameter sets aggregated weight for previous rounds Weighted covariance matrix is computed incrementally No need to store or access old objects and weights Efficiently computable from aggregated information Benefits: Guarantees closed query ellipsoids Suitable for high-dimensional multimedia data 9

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Relevance Feedback with Foresight (1) 10 Framework can be reused to tackle another challenge Exploratory search: user navigates through DB User picks objects to move query point into preferred direction New search region might be oriented contrary to intended movement Slow or no advancement Idea: Introduce heuristic direction matrix

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Relevance Feedback with Foresight (2) Orientation of matrix D depends on direction of query point movement Influence as a function of magnitude of movement Adjust seamlessly to phases of exploration and stationary refinement 11

Ranking Multimedia Databases via Relevance Feedback with History and Foresight Support / 12 Observations and Outlook Preliminary results Implemented prototype Relevance Feedback system History approach successfully extends MindReader to high dimensions Foresight promising but naïve functions sometimes showed too rapid or too slow a change in influence Work in progress: Suitable function for Foresight parameter Heuristics for aggregating Relevance Feedback results into gd Find gd using signature-based Relevance Feedback 12