Download presentation

Presentation is loading. Please wait.

Published byBertina Daniels Modified over 2 years ago

1
Fast N-Body Algorithms for Massive Datasets Alexander Gray Georgia Institute of Technology

2
Is science in 2007 different from science in 1907? Instruments [Science, Szalay & J. Gray, 2001]

3
Is science in 2007 different from science in 1907? 1990 COBE 1,000 2000 Boomerang 10,000 2002 CBI 50,000 2003 WMAP 1 Million 2008 Planck 10 Million Data: CMB Maps Data: Local Redshift Surveys 1986 CfA 3,500 1996 LCRS 23,000 2003 2dF 250,000 2005 SDSS 800,000 Data: Angular Surveys 1970 Lick 1M 1990 APM 2M 2005 SDSS 200M 2008 LSST 2B Instruments [Science, Szalay & J. Gray, 2001]

4
Sloan Digital Sky Survey (SDSS)

5
Size matters! Now possible: low noise: subtle patterns global properties and patterns rare objects and patterns more info: 3d, deeper/earlier, bands in parallel: more accurate simulations 2008: LSST – time-varying phenomena 1 billion objects 144 dimensions (~250M galaxies in 5 colors, ~1M 2000-D spectra)

6
Happening everywhere! Molecular biology microarray chips Earth sciences satellite topography Neuroscience functional MRI microprocessor nuclear mag. resonanceDrug discovery Physical simulation Internet fiber optics

7
1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? Astrophysicist Machine learning/ statistics guy R. Nichol, Inst. Cosmol. Gravitation A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Kulkarni, Inst. Cosmol. Gravitation D. Wake, Inst. Cosmol. Gravitation R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astronomy G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics

8
1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? Astrophysicist Machine learning/ statistics guy O(N n ) O(N 2 ) O(N 3 ) O(c D T(N)) R. Nichol, Inst. Cosmol. Grav. A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Kulkarni, Inst. Cosmol. Grav. D. Wake, Inst. Cosmol. Grav. R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astro. G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics Kernel density estimator n-point spatial statistics Nonparametric Bayes classifier Support vector machine Nearest-neighbor statistics Gaussian process regression Bayesian inference

9
R. Nichol, Inst. Cosmol. Grav. A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Kulkarni, Inst. Cosmol. Grav. D. Wake, Inst. Cosmol. Grav. R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astro. G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics Kernel density estimator n-point spatial statistics Nonparametric Bayes classifier Support vector machine Nearest-neighbor statistics Gaussian process regression Bayesian inference 1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? Astrophysicist Machine learning/ statistics guy O(N n ) O(N 2 ) O(N 3 ) O(c D T(N))

10
R. Nichol, Inst. Cosmol. Grav. A. Connolly, U. Pitt Physics C. Miller, NOAO R. Brunner, NCSA G. Kulkarni, Inst. Cosmol. Grav. D. Wake, Inst. Cosmol. Grav. R. Scranton, U. Pitt Physics M. Balogh, U. Waterloo Physics I. Szapudi, U. Hawaii Inst. Astro. G. Richards, Princeton Physics A. Szalay, Johns Hopkins Physics Kernel density estimator n-point spatial statistics Nonparametric Bayes classifier Support vector machine Nearest-neighbor statistics Gaussian process regression Bayesian inference 1.How did galaxies evolve? 2.What was the early universe like? 3.Does dark energy exist? 4.Is our model (GR+inflation) right? But I have 1 million points Astrophysicist Machine learning/ statistics guy O(N n ) O(N 2 ) O(N 3 ) O(c D T(N))

11
Data: The Stack Apps (User, Science) Perception Computer Vision, NLP, Machine Translation, Bibleome, Autonomous vehicles ML / Opt Machine Learning / Optimization / Linear Algebra / Privacy Data Abstractions DBMS, MapReduce, VOTables, Clustering / Threading Programming with 1000s of powerful compute nodes O/S Network Motherboards / Datacenter ICs

12
Data: The Stack Apps (User, Science) Perception Computer Vision, NLP, Machine Translation, Bibleome, Autonomous vehicles ML / Opt Machine Learning / Optimization / Linear Algebra / Privacy Data Abstractions DBMS, MapReduce, VOTables, Data structures Clustering / Threading Programming with 1000s of powerful compute nodes O/S Network Motherboards / Datacenter ICs

13
Making fast algorithms There are many large datasets. There are many questions we want to ask them. –Why we must not get obsessed with one specific dataset. –Why we must not get obsessed with one specific question. The activity I’ll describe is about accerating computations which occur commonly across many ML methods.

14
Scope Nearest neighbor K-means Hierarchical clustering N-point correlation functions Kernel density estimation Locally-weighted regression Mean shift tracking Mixtures of Gaussians Gaussian process regression Manifold learning Support vector machines Affinity propagation PCA ….

15
Scope ML methods with distances underneath –Distances only –Continuous kernel functions ML methods with counting underneath

16
Scope Computational ideas in this tutorial: –Data structures –Monte Carlo –Series expansions –Problem/solution abstractions Challenges –Don’t introduce error, if possible –Don’t introduce tweak parameters, if possible

17
Two canonical problems Nearest-neighbor search Kernel density estimation

18
Ideas 1.Data structures and how to use them 2.Monte Carlo 3.Series expansions 4.Problem/solution abstractions

19
Slides by Jeremy Kubica Nearest Neighbor - Naïve Approach Given a query point X. Scan through each point Y: –Calculate the distance d(X,Y) –If d(X,Y) < best_seen then Y is the new nearest neighbor. Takes O(N) time for each query! 33 Distance Computations

20
Slides by Jeremy Kubica Speeding Up Nearest Neighbor We can speed up the search for the nearest neighbor: –Examine nearby points first. –Ignore any points that are further then the nearest point found so far. Do this using a KD-tree: –Tree based data structure –Recursively partitions points into axis aligned boxes.

21
Slides by Jeremy Kubica KD-Tree Construction PtXY 10.00 21.004.31 30.132.85 ……… We start with a list of n-dimensional points.

22
Slides by Jeremy Kubica KD-Tree Construction PtXY 10.00 30.132.85 ……… We can split the points into 2 groups by choosing a dimension X and value V and separating the points into X > V and X <= V. X>.5 PtXY 21.004.31 ……… YES NO

23
Slides by Jeremy Kubica KD-Tree Construction PtXY 10.00 30.132.85 ……… We can then consider each group separately and possibly split again (along same/different dimension). X>.5 PtXY 21.004.31 ……… YES NO

24
Slides by Jeremy Kubica KD-Tree Construction PtXY 30.132.85 ……… We can then consider each group separately and possibly split again (along same/different dimension). X>.5 PtXY 21.004.31 ……… YES NO PtXY 10.00 ……… Y>.1 NO YES

25
Slides by Jeremy Kubica KD-Tree Construction We can keep splitting the points in each set to create a tree structure. Each node with no children (leaf node) contains a list of points.

26
Slides by Jeremy Kubica KD-Tree Construction We will keep around one additional piece of information at each node. The (tight) bounds of the points at or below this node.

27
Slides by Jeremy Kubica KD-Tree Construction Use heuristics to make splitting decisions: Which dimension do we split along? Widest Which value do we split at? Median of value of that split dimension for the points. When do we stop? When there are fewer then m points left OR the box has hit some minimum width.

28
Slides by Jeremy Kubica Exclusion and inclusion, using point-node kd-tree bounds. O(D) bounds on distance minima/maxima:

29
Slides by Jeremy Kubica Exclusion and inclusion, using point-node kd-tree bounds. O(D) bounds on distance minima/maxima:

30
Slides by Jeremy Kubica Nearest Neighbor with KD Trees We traverse the tree looking for the nearest neighbor of the query point.

31
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Examine nearby points first: Explore the branch of the tree that is closest to the query point first.

32
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Examine nearby points first: Explore the branch of the tree that is closest to the query point first.

33
Slides by Jeremy Kubica Nearest Neighbor with KD Trees When we reach a leaf node: compute the distance to each point in the node.

34
Slides by Jeremy Kubica Nearest Neighbor with KD Trees When we reach a leaf node: compute the distance to each point in the node.

35
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Then we can backtrack and try the other branch at each node visited.

36
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Each time a new closest node is found, we can update the distance bounds.

37
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.

38
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.

39
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.

40
Slides by Jeremy Kubica Simple recursive algorithm (k=1 case) NN(x q,R,d lo,x sofar,d sofar ) { if d lo > d sofar, return. if leaf(R), [x sofar,d sofar ]=NNBase(x q,R,d sofar ). else, [R1,d1,R2,d2]=orderByDist(x q,R.l,R.r). NN(x q,R1,d1,x sofar,d sofar ). NN(x q,R2,d2,x sofar,d sofar ). }

41
Slides by Jeremy Kubica Nearest Neighbor with KD Trees Instead, some animations showing real data… 1.kd-tree with cached sufficient statistics 2.nearest-neighbor with kd-trees 3.range-count with kd-trees For animations, see: http://www.cs.cmu.edu/~awm/animations/kdtree

42
Range-count example

46
Pruned! (inclusion)

47
Range-count example

54
Pruned! (exclusion)

55
Range-count example

58
Some questions Asymptotic runtime analysis? –In a rough sense, O(logN) –But only under some regularity conditions How high in dimension can we go? –Roughly exponential in intrinsic dimension –In practice, in less than 100 dimensions, still big speedups

59
Another kind of tree Ball-trees, metric trees –Use balls instead of hyperrectangles –Can often be more efficient in high dimension (though not always) –Can work with non-Euclidean metric (you only need to respect the triangle inequality) –Many non-metric similarity measures can be bounded by metric quantities.

60
A Set of Points in a metric space

61
Ball Tree root node

62
A Ball Tree

66
J. Uhlmann, 1991 S. Omohundro, NIPS 1991

67
Ball-trees: properties Let Q be any query point and let x be a point inside ball B |x-Q| |Q - B.center| - B.radius |x-Q| |Q - B.center| + B.radius Q B.center x

68
How to build a metric tree, exactly? Must balance quality vs. build-time ‘Anchors hierarchy’ (farthest-points heuristic, 2-approx used in OR) Omohundro: ‘Five ways to build a ball- tree’ Which is the best? A research topic…

69
Some other trees Cover-tree –Provable worst-case O(logN) under an assumption (bounded expansion constant) –Like a non-binary ball-tree Learning trees –In this conference

70
‘All’-type problems Nearest-neighbor search All-nearest neighbor (bichromatic): Kernel density estimation ‘All’ version (bichromatic):

71
Almost always ‘all’-type problems Kernel density estimation Nadaraya-Watson & locally-wgtd regression Gaussian process prediction Radial basis function networks Monochromatic all-nearest neighbor (e.g. LLE) n-point correlation (n-tuples) Always ‘all’-type problems

72
Dual-tree idea If all the queries are available simultaneously, then it is faster to: 1. Build a tree on the queries as well 2. Effectively process the queries in chunks rather than individually work is shared between similar query points

73
Single-tree:

74
Dual-tree (symmetric):

75
Exclusion and inclusion, using point-node kd-tree bounds. O(D) bounds on distance minima/maxima:

76
Exclusion and inclusion, using point-node kd-tree bounds. O(D) bounds on distance minima/maxima:

77
Exclusion and inclusion, using kd-tree node-node bounds. O(D) bounds on distance minima/maxima: (Analogous to point-node bounds.) Also needed: Nodewise bounds.

78
Exclusion and inclusion, using kd-tree node-node bounds. O(D) bounds on distance minima/maxima: Also needed: (Analogous to point-node bounds.) Nodewise bounds.

79
Single-tree: simple recursive algorithm (k=1 case) NN(x q,R,d lo,x sofar,d sofar ) { if d lo > d sofar, return. if leaf(R), [x sofar,d sofar ]=NNBase(x q,R,d sofar ). else, [R1,d1,R2,d2]=orderByDist(x q,R.l,R.r). NN(x q,R1,d1,x sofar,d sofar ). NN(x q,R2,d2,x sofar,d sofar ). }

80
Single-tree Dual-tree x q Q d lo (x q,R) d lo (Q,R) x sofar x sofar, d sofar d sofar store Q.d sofar =max Q d sofar 2-way recursion 4-way recursion N x O(logN) O(N)

81
Dual-tree: simple recursive algorithm (k=1) AllNN(Q,R,d lo,x sofar,d sofar ) { if d lo > Q.d sofar, return. if leaf(Q) & leaf(R), [x sofar,d sofar ]=AllNNBase(Q,R,d sofar ). Q.d sofar =max Q d sofar. else if !leaf(Q) & leaf(R), … else if leaf(Q) & !leaf(R), … else if !leaf(Q) & !leaf(R), [R1,d1,R2,d2]=orderByDist(Q.l,R.l,R.r). AllNN(Q.l,R1,d1,x sofar,d sofar ). AllNN(Q.l,R2,d2,x sofar,d sofar ). [R1,d1,R2,d2]=orderByDist(Q.r,R.l,R.r). AllNN(Q.r,R1,d1,x sofar,d sofar ). AllNN(Q.r,R2,d2,x sofar,d sofar ). Q.d sofar = max(Q.l.d sofar,Q.r.d sofar ). }

82
Query points Reference points Dual-tree traversal (depth-first)

83
Query points Reference points Dual-tree traversal

84
Query points Reference points Dual-tree traversal

85
Query points Reference points Dual-tree traversal

86
Query points Reference points Dual-tree traversal

87
Query points Reference points Dual-tree traversal

88
Query points Reference points Dual-tree traversal

89
Query points Reference points Dual-tree traversal

90
Query points Reference points Dual-tree traversal

91
Query points Reference points Dual-tree traversal

92
Query points Reference points Dual-tree traversal

93
Query points Reference points Dual-tree traversal

94
Query points Reference points Dual-tree traversal

95
Query points Reference points Dual-tree traversal

96
Query points Reference points Dual-tree traversal

97
Query points Reference points Dual-tree traversal

98
Query points Reference points Dual-tree traversal

99
Query points Reference points Dual-tree traversal

100
Query points Reference points Dual-tree traversal

101
Query points Reference points Dual-tree traversal

102
Query points Reference points Dual-tree traversal

103
Meta-idea: Higher-order Divide-and-conquer Break each set into pieces. Solving the sub-parts of the problem and combining these sub-solutions appropriately might be easier than doing this over only one set. Generalizes divide-and-conquer of a single set to divide-and-conquer of multiple sets.

104
Ideas 1.Data structures and how to use them 2.Monte Carlo 3.Series expansions 4.Problem/solution abstractions

105
2-point correlation r Characterization of an entire distribution? “How many pairs have distance < r ?” 2-point correlation function

106
The n-point correlation functions Spatial inferences: filaments, clusters, voids, homogeneity, isotropy, 2-sample testing, … Foundation for theory of point processes [Daley,Vere-Jones 1972], unifies spatial statistics [Ripley 1976] Used heavily in biostatistics, cosmology, particle physics, statistical physics 2pcf definition: 3pcf definition:

107
3-point correlation “How many triples have pairwise distances < r ?” r3r3 r1r1 r2r2 Standard model: n>0 terms should be zero!

108
How can we count n-tuples efficiently? “How many triples have pairwise distances < r ?”

109
Use n trees! [Gray & Moore, NIPS 2000]

110
“How many valid triangles a-b-c (where ) could there be? A B C r count{A,B,C} = ?

111
“How many valid triangles a-b-c (where ) could there be? count{A,B,C} = count{A,B, C.left } + count{A,B,C.right} A B C r

112
“How many valid triangles a-b-c (where ) could there be? A B C r count{A,B,C} = count{A,B,C.left} + count{A,B, C.right }

113
“How many valid triangles a-b-c (where ) could there be? A B C r count{A,B,C} = ?

114
“How many valid triangles a-b-c (where ) could there be? A B C r Exclusion count{A,B,C} = 0!

115
“How many valid triangles a-b-c (where ) could there be? AB C count{A,B,C} = ? r

116
“How many valid triangles a-b-c (where ) could there be? AB C Inclusion count{A,B,C} = |A| x |B| x |C| r Inclusion

117
Key idea (combinatorial proximity problems): for n-tuples: n-tree recursion

118
3-point runtime (biggest previous: 20K) VIRGO simulation data, N = 75,000,000 naïve: 5x10 9 sec. (~150 years) multi-tree: 55 sec. (exact) n=2: O(N) n=3: O(N log3 ) n=4: O(N 2 )

119
But… Depends on r D-1. Slow for large radii. VIRGO simulation data, N = 75,000,000 naïve: ~150 years multi-tree: large h: 24 hrs Let’s develop a method for large radii.

120
c = p T EASIER? known. hard. no dependence on N! but it does depend on p

121
c = p T no dependence on N! but it does depend on p

122
c = p T no dependence on N! but it does depend on p

123
c = p T no dependence on N! but it does depend on p

124
c = p T no dependence on N! but it does depend on p

125
This is junk: don’t bother c = p T

126
This is promising c = p T

127
Basic idea: 1. Remove some junk (Run exact algorithm for a while) make p larger 2. Sample from the rest

128
Get disjoint sets from the recursion tree … … … [prune] all possible n-tuples

129
+ + = + + = Now do stratified sampling

130
Speedup Results VIRGO simulation data N = 75,000,000 naïve: ~150 years multi-tree: large h: 24 hrs multi-tree monte carlo: 99% confidence: 96 sec

131
Ideas 1.Data structures and how to use them 2.Monte Carlo 3.Multipole methods 4.Problem/solution abstractions

132
Kernel density estimation

133
How to use a tree… 1. How to approximate? 2. When to approximate? [Barnes and Hut, Science, 1987] if s R r

134
How to use a tree… 3. How to know potential error? Let’s maintain bounds on the true kernel sum At the beginning: R

135
Single-tree: Dual-tree (symmetric): [Gray & Moore 2000] How to use a tree… 4. How to do ‘all’ problem?

136
rRrR rQrQ s if Generalizes Barnes-Hut to dual-tree RQ

137
Case 1 – alg. gives no error bounds Case 2 – alg. gives error bounds, but must be rerun Case 3 – alg. automatically achieves error tolerance BUT: We have a tweak parameter: So far we have case 2; let’s try for case 3 Let’s try to make an automatic stopping rule

138
Finite-difference function approximation. Taylor expansion: Gregory-Newton finite form:

139
Finite-difference function approximation. assumes monotonic decreasing kernel approximate {Q,R} if

140
Speedup Results (KDE) One order-of-magnitude speedup over single-tree at ~2M points 12.5K7.12 25K31.31 50K123.46 100K4941.0 200K1976*2 400K7904*5 800K31616*10 1.6M35 hrs23 dual- N naïve tree 5500x

141
Ideas 1.Data structures and how to use them 2.Monte Carlo 3.Multipole methods 4.Problem/solution abstractions

142
These are all examples of… Generalized N-body problems General theory and toolkit for designing algorithms for such problems All-NN: 2-point: 3-point: KDE: SPH:

143
For more… In this conference: Learning trees Monte Carlo for statistical summations Large-scale learning workshop EMST GNP’s and MapReduce-like parallelization Monte Carlo SVD agray@cc.gatech.edu

Similar presentations

OK

1 Efficient Algorithms for Non-parametric Clustering With Clutter Weng-Keen Wong Andrew Moore (In partial fulfillment of the speaking requirement)

1 Efficient Algorithms for Non-parametric Clustering With Clutter Weng-Keen Wong Andrew Moore (In partial fulfillment of the speaking requirement)

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google