1. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20011 Indexing Images with Multiple Regions Euripides G.M. Petrakis http://www.ced.tuc.gr/~petrakis Dept. of Electronic and Computer Engineering Technical University of Crete (TUC) Chania, Crete, Greece

2. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20012 Problem Definition Given a database with N images  Each image may contain more than one object. Retrieve images similar to a query Q:  Similar objects;  Similar spatial relationships. Respond faster than sequential scanning for  D(Q,I) <= t (range queries);  Retrieve the k most similar images (NN queries).

3. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20013 Contributions We formulate the problem of indexing images with multiple regions as one of spatial searching using Spatial Access Methods (SAMs) like e.g., R-trees. We show how a SAM can be used to treat images with multiple objects and answer:  Nearest Neighbor queries;  Range queries. Two algorithms are proposed, one for each type of query.

4. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20014 Indexing Approach Assumption: Each object is represented by an n- dimensional feature vector (v 1 v 2 …v n ):  E.g., (size, orientation, roundness, color, texture);  Usually low dimensional vector;  Distance between objects D f : any vector distance like Euclidean, Manhattan etc. Map each vector to a n-dimensional feature space:  Each object  one point;  Image (query) with M objects  M points. Apply a SAM for indexing (R-tree, SR-tree etc).

5. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20015 Mapping images I=(I 1,I 2, I 3 ) and J=(J 1,J 2 ) and query Q=(Q 1,Q 2 ) Q1Q1 Q 2 I1I1 I2I2 I3I3 J1J1 J2J2 t t size roundnessroundness

6. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20016 Problems with SAMs Existing algorithms can treat range or NN queries for each Q 1 or Q 2 but not for Q as a whole  E.g., find the k –NNs of Q 1 or Q 2 ;  Similarly for range queries. A SAM retrieves the k-NNs with respect to D f not to D (distance between whole images)  D = function (D f );  Similarly for range queries.

7. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20017 Range Queries Input: query Q, distances D, D f, tolerance t. Output: images I satisfying D(Q,I) <= t. 1.Decompose Q into Q 1,Q 2,…,Q m queries; 2.Apply D f (Q i,I j ) <= t  store results in A i ; 3.Compute ; 4.For each I in A compute D(Q,I); 5.Output images satisfying D(Q,I) <= t;

8. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20018 Nearest Neighbor (NN) Queries Input: query Q, distance D, D f, number k. Output: the k images most similar to Q. 1.Decompose Q into Q 1,Q 2,…,Q m queries; 2.Apply a k-NN query for each Q i. Retrieve k distinct images (incremental k-NN search); Compute t i, : max D of the k NNs of Q i from Q; 3.Compute t = min{t i }. 4.Apply a range query D(Q,I) <= t. 5.Output the k images closest to Q.

9. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 20019 Comments on the Two Algorithms Assumption: image distance satisfies the “Lower Bounding Principle” D f (Q,I) <= D(Q,I).  Careful design of image distances is required;  No false dismissals or false drops. The performance depends on t: the lower the t the faster the algorithms are. NN queries are slower than range queries; Optimization: do not apply all Q i ’s.

10. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200110 Definition Image Distance (1) Image matching as an assignment problem (Hungarian algorithm). D(Q,I) : cost of the best mapping of objects of Q to objects in I. Cost of a mapping. C(Q,I) = Σ D f (i,j). D(Q,I) = min { C(Q,I) }. D f (Q,I) <= D(Q,I) ! Ignores relationships. DfDf D(Q,I) = 10

11. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200111 Experiments Dataset of 13,500 synthetic images.  each image contains 4-8 objects;  90,000 objects (vectors) are stored in an R-tree; The results are averages over 20 queries. Demonstrate the superiority of the proposed approach over sequential scan searching.

12. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200112 Speed-up: Range Queries

13. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200113 Speed-up: NN queries

14. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200114 Conclusions Interesting problem, many applications:  image, video retrieval, data mining etc. Disadvantages of the proposed solution:  Suitable for “small” images (up to 8 objects);  Needs incremental NN search. Future work:  More efficient algorithms ?

15. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200115 Definition of Image Distance (2) Image matching as a transformation of the ARG of I to the ARG of Q (A* algorithm).  D(Q,I): minimum cost transformation. Cost of a transformation C(Q,I) = max { D f (i,j) }. D f (Q,I) <= D(Q,I)!

16. Euripides G.M. PetrakisIR'2001 Oulu, 19-21 Sept. 200116 Retrieval Example