Euripides G.M. PetrakisIR'2001 Oulu, Sept Indexing Images with Multiple Regions Euripides G.M. Petrakis Dept. of Electronic and Computer Engineering Technical University of Crete (TUC) Chania, Crete, Greece

Euripides G.M. PetrakisIR'2001 Oulu, Sept Problem Definition Given a database with N images  Each image may contain more than one object. Retrieve images similar to a query Q:  Similar objects;  Similar spatial relationships. Respond faster than sequential scanning for  D(Q,I) <= t (range queries);  Retrieve the k most similar images (NN queries).

Euripides G.M. PetrakisIR'2001 Oulu, Sept Contributions We formulate the problem of indexing images with multiple regions as one of spatial searching using Spatial Access Methods (SAMs) like e.g., R-trees. We show how a SAM can be used to treat images with multiple objects and answer:  Nearest Neighbor queries;  Range queries. Two algorithms are proposed, one for each type of query.

Euripides G.M. PetrakisIR'2001 Oulu, Sept Indexing Approach Assumption: Each object is represented by an n- dimensional feature vector (v 1 v 2 …v n ):  E.g., (size, orientation, roundness, color, texture);  Usually low dimensional vector;  Distance between objects D f : any vector distance like Euclidean, Manhattan etc. Map each vector to a n-dimensional feature space:  Each object  one point;  Image (query) with M objects  M points. Apply a SAM for indexing (R-tree, SR-tree etc).

Euripides G.M. PetrakisIR'2001 Oulu, Sept Mapping images I=(I 1,I 2, I 3 ) and J=(J 1,J 2 ) and query Q=(Q 1,Q 2 ) Q1Q1 Q 2 I1I1 I2I2 I3I3 J1J1 J2J2 t t size roundnessroundness

Euripides G.M. PetrakisIR'2001 Oulu, Sept Problems with SAMs Existing algorithms can treat range or NN queries for each Q 1 or Q 2 but not for Q as a whole  E.g., find the k –NNs of Q 1 or Q 2 ;  Similarly for range queries. A SAM retrieves the k-NNs with respect to D f not to D (distance between whole images)  D = function (D f );  Similarly for range queries.

Euripides G.M. PetrakisIR'2001 Oulu, Sept Range Queries Input: query Q, distances D, D f, tolerance t. Output: images I satisfying D(Q,I) <= t. 1.Decompose Q into Q 1,Q 2,…,Q m queries; 2.Apply D f (Q i,I j ) <= t  store results in A i ; 3.Compute ; 4.For each I in A compute D(Q,I); 5.Output images satisfying D(Q,I) <= t;

Euripides G.M. PetrakisIR'2001 Oulu, Sept Nearest Neighbor (NN) Queries Input: query Q, distance D, D f, number k. Output: the k images most similar to Q. 1.Decompose Q into Q 1,Q 2,…,Q m queries; 2.Apply a k-NN query for each Q i. Retrieve k distinct images (incremental k-NN search); Compute t i, : max D of the k NNs of Q i from Q; 3.Compute t = min{t i }. 4.Apply a range query D(Q,I) <= t. 5.Output the k images closest to Q.

Euripides G.M. PetrakisIR'2001 Oulu, Sept Comments on the Two Algorithms Assumption: image distance satisfies the “Lower Bounding Principle” D f (Q,I) <= D(Q,I).  Careful design of image distances is required;  No false dismissals or false drops. The performance depends on t: the lower the t the faster the algorithms are. NN queries are slower than range queries; Optimization: do not apply all Q i ’s.

Euripides G.M. PetrakisIR'2001 Oulu, Sept Definition Image Distance (1) Image matching as an assignment problem (Hungarian algorithm). D(Q,I) : cost of the best mapping of objects of Q to objects in I. Cost of a mapping. C(Q,I) = Σ D f (i,j). D(Q,I) = min { C(Q,I) }. D f (Q,I) <= D(Q,I) ! Ignores relationships. DfDf D(Q,I) = 10

Euripides G.M. PetrakisIR'2001 Oulu, Sept Experiments Dataset of 13,500 synthetic images.  each image contains 4-8 objects;  90,000 objects (vectors) are stored in an R-tree; The results are averages over 20 queries. Demonstrate the superiority of the proposed approach over sequential scan searching.

Euripides G.M. PetrakisIR'2001 Oulu, Sept Speed-up: Range Queries

Euripides G.M. PetrakisIR'2001 Oulu, Sept Speed-up: NN queries

Euripides G.M. PetrakisIR'2001 Oulu, Sept Conclusions Interesting problem, many applications:  image, video retrieval, data mining etc. Disadvantages of the proposed solution:  Suitable for “small” images (up to 8 objects);  Needs incremental NN search. Future work:  More efficient algorithms ?

Euripides G.M. PetrakisIR'2001 Oulu, Sept Definition of Image Distance (2) Image matching as a transformation of the ARG of I to the ARG of Q (A* algorithm).  D(Q,I): minimum cost transformation. Cost of a transformation C(Q,I) = max { D f (i,j) }. D f (Q,I) <= D(Q,I)!

Euripides G.M. PetrakisIR'2001 Oulu, Sept Retrieval Example