1. Indexing Multidimensional Data Rui Zhang http://www.csse.unimelb.edu.au/~rui The University of Melbourne Aug 2006

2. Outline Backgrounds Multidimensional data and queries Approaches Mapping based indexing Z-curve iDistance Hierarchical-tree based indexing R-tree k-d-tree Quad-tree Compression based indexing VA-file

3. Multidimensional Data Spatial data Geographic Information: Melbourne (37, 145) Which city is at (30, 140)? Computer Aided Design: width and height (40, 50) Any part that has a width of 40 and height of 50? Records with multiple attributes Employee (ID, age, score, salary, …) Is there any employee whose age is under 25 and performance score is greater than 80 and salary is between 3000 and 5000 Multimedia data Color histograms of images Give me the most similar image to Multimedia Features: color, shape, texture IDAgeScoreSalary… … (high-dimensionality) (medium-dimensionality) (low-dimensionality)

4. Multidimensional Queries Point query Return the objects located at Q(x 1, x 2, …, x d ). E.g. Q=(3.4, 6.6). Window query Return all the objects enclosed or intersected by the hyper-rectangle W{[L 1, U 1 ], [L 2, U 2 ], …, [L d, U d ]}. E.g. W={[0,4],[2,5]} K-Nearest Neighbor Query (KNN Query) Return k objects whose distances to Q are no larger than any other object’ distance to Q. E.g. 3NN of Q=(4,1)

5. Mapping Based Multidimensional Indexing Story The CBD: [0,4][2,5] Blocks in the CBD are: [8,15], [32,33] and [36,37] General strategy: three steps Data mapping and indexing Query mapping and data retrieval Filtering out false positive NamexyBlockHeight A0.71.22100 B5.81.21950 C2.72.31280 D5.52.42590 E6.62.52840 F1.73.811120 G2.84.736100 H0.65.83450 I1.66.74160 J3.46.64540 NamexyBlockHeight A0.71.22100 F1.73.811120 C2.72.31280 B5.81.21950 D5.52.42590 E6.62.52840 H0.65.83450 G2.84.736100 I1.66.74160 J3.46.64540 Sort

6. The Z-curve and Other Space-Filling Curves The Z-curve Z-value calculation: bit-interleaving Support efficient window queries Disadvantage Jumps Other space-filling curves Hilbert-curves Gray-code Column-wise scan

7. 3 2 1 Mapping for KNN Queries Story continued New factory at Q[4,1] Find 3 nearest buildings to Q Termination condition K candidates All in the current search circle NamexyStreetHeight A0.71.214100 B5.81.23250 C2.72.31280 D5.52.43190 E6.62.53240 F1.73.813120 G2.84.724100 H0.65.82350 I1.66.72260 J3.46.62440 Sort 11 12 13 14 21 22 23 24 31 32 NamexyStreetHeight C2.72.31280 F1.73.813120 A0.71.214100 I1.66.72260 H0.65.82350 G2.84.724100 J3.46.62440 D5.52.43190 B5.81.23250 E6.62.53240 Rank123 CandidateA Distance to Q3.31 Q Rank123 CandidateBAF Distance to Q1.813.313.62 Rank123 CandidateBEA Distance to Q1.813.003.31 Rank123 CandidateAF Distance to Q3.313.62 Rank123 CandidateBCE Distance to Q1.811.843.00 Rank123 CandidateBCD Distance to Q1.811.842.05 ||AQ|| = 3.31||FQ|| = 3.62||BQ|| = 1.81||EQ|| = 3.00||CQ|| = 1.84||DQ|| = 2.05 1 2 3 4 R = 0.35R = 0.70R = 1.05R = 1.40R = 1.75R = 2.10

8. The iDistance Data partitioned into a number of clusters Streets are concentric circles Data mapping Objects mapped to street numbers Query mapping Search circle mapped to streets intersected

9. Hierarchical Tree Structures R-tree Minimum bounding rectangle (MBR) Incomplete and overlapping partitioning Disk-based; Balanced A D C E B F G A D C E B F G A D C E B G F A D C E B G F K-d-tree Space division recursively Complete and disjoint partitioning In-memory; Unbalanced There are algorithms to page and balance the tree, but with more complex manipulations A N1N1 N2N2 N1N1 BCD N1N1 ACD N1N1 BE N2N2 N1N1 N2N2 FG N1N1 N3N3 N3N3 ABCD N1N1 0.5 N3N3 N1N1 N2N2 AD N1N1 BCE N2N2 N3N3 F BCE N2N2 FG N4N4 N4N4 N5N5 0.3 N5N5 Problem: OverlapProblem: Empty space

10. Hierarchical Tree Structures (continued) Quad-tree Space divided into 4 rectangles recursively. Complete and disjoint partitioning In-memory; Unbalanced There are algorithms to page and balance the tree, but with more complex manipulations The point quad-tree A D C E B F G A NW NE SW B NW SWSE NE CD EFG SE

11. Compression Based Indexing The dimensionality curse The Vector Approximation File (VA-File) VA FileSkewed data

12. Summary of the Indexing Techniques IndexDisk-based / In-memory BalancedEfficient query type Dimensi onality Comments R-treeDisk-basedYesPoint, window, kNN LowDisadvantage is overlap K-d-treeIn-memoryNoPoint, window, kNN(?) LowInefficient for skewed data Quad-treeIn-memoryNoPoint, window, kNN(?) LowInefficient for skewed data Z-curve + B + -tree Disk-basedYesPoint, window LowOrder of the Z- curve affects performance iDistanceDisk-basedYesPoint, kNNHighNot good for uniform data in very high-D VA-FileDisk-basedPoint, window, kNN HighNot good for skewed data

13. Index Implementations in major DBMS SQL Server B+-Tree data structure Clustered indexes are sparse Indexes maintained as updates/insertions/deletes are performed Oracle B+-tree, hash, bitmap, spatial extender for R-Tree Clustered index Index organized table (unique/clustered) Clusters used when creating tables DB2 B+-Tree data structure, spatial extender for R-tree Clustered indexes are dense Explicit command for index reorganization

14. Recommended Readings and References Survey on multidimensional indexing techniques Christian B ö hm, Stefan Berchtold, Daniel A. Keim. Searching in high-dimensional spaces: Index structures for improving the performance of multimedia databases. ACM Computing Surveys 2001. Volker Gaede, Oliver G ü nther. Multidimensional Access Methods. ACM Computing Surveys 1998 Mapping based indexing Rui Zhang, Panos Kalnis, Beng Chin Ooi, Kian-Lee Tan. Generalized Multi-dimensional Data Mapping and Query Processing. ACM Transactions on Data Base Systems (TODS), 30(3), 2005. Space-filling curves H. V. Jagadish. Linear Clustering of Objects with Multiple Atributes. ACM SIGMOD Conference (SIGMOD) 1990. iDistance H.V. Jagadish, Beng Chin Ooi, Kian-Lee Tan, Cui Yu, Rui Zhang. iDistance: An Adaptive B+-tree Based Indexing Method for Nearest Neighbor Search. ACM Transactions on Data Base Systems (TODS), 30(2), 2005. R-tree Antonin Guttman. R-Trees: A Dynamic Index Structure for Spatial Searching. ACM SIGMOD Conference (SIGMOD) 1984. Quad-tree Hanan Samet. The Quadtree and Related Hierarchical Data Structures. ACM Computing Surveys 1984. VA-File Roger Weber, Hans-J ö rg Schek, Stephen Blott. A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces. International Conference on Very Large Data Bases (VLDB) 1998.