When Is Nearest Neighbors Indexable? Uri Shaft (Oracle Corp.) Raghu Ramakrishnan (UW-Madison)

Motivation - Scalability Experiments Dozens of papers describe experiments about index scalability with increased dimensions. –Constants are: Number of data points Data and Query distribution Index structure / search algorithm –Variable: Number of dimensions –Measurement: Performance of index.

Example From PODS 1997

Motivation In many cases the conclusion is that the empirical evidence suggests the index structures do scale with dimensionality We would like to investigate these claims mathematically – supply a proof of scalability or non-scalability.

Historical Context Continues work done in When Is Nearest Neighbors Meaningful? (ICDT 1999) Previous work about behavior of distance distributions. This work about behavior of indexing structures under similar conditions.

Contents Vanishing Variance property Convex Description index structures Indexing Theorem –The performance of CD index does not scale for VV workloads using Euclidean distances. Conclusion Future Work

Vanishing Variance Same definition used in ICDT 99 work (although not named in that work) In 1999 we showed that the workloads become meaningless – ratios of distances between query and various data points become arbitrarily small. We use the same result here.

Vanishing Variance A scalability experiment contains a series of workloads W 1,W 2,…,W m,… –m is the number of dimensions –each workload W 1 has n data points and a query point (same distribution) –Distance distribution marked as D m Vanishing Variance:

Contents Vanishing Variance property Convex Description index structures Indexing Theorem –The performance of CD index does not scale for VV workloads using Euclidean distances. Conclusion Future Work

Convex Description Index Data points distributed to buckets (e.g. disk pages). Access to a buckets is all or nothing. We allow redundancy. A bucket contains at least two data points. Each bucket associated with a description – a convex region containing all data points in the bucket. Search algorithm accesses at least all buckets whose convex region is closer than the nearest neighbor. Cost of search is the number of data points retrieved.

Example: R-Tree Buckets are disk pages. Under normal construction buckets contain more than two data points each. Bucket descriptions are convex and contain all data points (Bounding Rectangles). Search algorithm accesses all buckets whose convex region is closer than the nearest neighbor (and probably a few more).

Convex Description Indexes All R-Tree variants X-Tree M-Tree kdb-Tree SS-Tree and SR-Tree Many more

Other indexes (non-CD) Probability structures (P-Tree, VLDB 2000) –Access based on clusters. A near enough bucket may not be accessed Projection index (like VA-file) –Compression structures. –All data points accessed in pieces, not in buckets.

Contents Vanishing Variance property Convex Description index structures Indexing Theorem –The performance of CD index does not scale for VV workloads using Euclidean distances. Conclusion Future Work

Indexing Theorem If: –Scalability experiment uses a series of workloads with Vanishing Variance –The distance metric is Euclidean –The indexing structure is Convex Description Then: –The expected cost of a query converges to the number of data points – I.e., a linear scan of the data

Sketch of Proof Because of Vanishing Variance, the ratio of distances between various query and data points becomes arbitrarily close to 1. When using Euclidean distance, we can look at an arbitrary data bucket and a query point, choose two data points from the bucket and create a triangle:

Bucket Q D1D1 D2D2 Y Distances of Q, D 1, D 2,…, D n are about the same. Distance of Q to Y is much smaller Therefore, distance of Q to data bucket is less than distance to nearest neighbor.

Contents Vanishing Variance property Convex Description index structures Indexing Theorem –The performance of CD index does not scale for VV workloads using Euclidean distances. Conclusion Future Work

Conclusion Dozens of papers describe experiments about index scalability with increased dimensions. We wanted to investigate these claims mathematically – supply a proof of scalability or non-scalability. We proved that many of these experiments do not scale in dimensionality.

Conclusion Use this theorem to to channel indexing research into more useful and practical avenues Review previous results accordingly.

Future Work Remove restriction of at least two data points in bucket. –Easy exercise, need to take into account the cost of traversing a hierarchical data structure. Investigate other L p metrics Investigate projection indexes using Euclidean metric (looks like they do not scale either)

Find scalable indexing structure for Uniform data and L metric –Hint: use compression Find number of data points needed for R-Tree to be practical on uniform data, L 2 metric. –Approx: Future Work

Questions