Presentation on theme: "Simplicial Depth: An Improved Definition, Analysis, and Efficiency in the Finite Sample Case Michael A. Burr, Eynat Rafalin, and Diane L. Souvaine Tufts."— Presentation transcript:
Simplicial Depth: An Improved Definition, Analysis, and Efficiency in the Finite Sample Case Michael A. Burr, Eynat Rafalin, and Diane L. Souvaine Tufts University CCCG 2004NSF grant #EIA
Introduction Introduction to Data Depth –Why? –Examples –Desirable Properties Simplicial Depth –Definition –Properties –Problems Revised Definition –Definition –Properties Ongoing work
What is Data Depth and Why? Measures how deep (central) a given point is relative to a distribution or a data cloud. –Deals with the shape of the data. –Can be thought of as a measure of how well a point characterizes a data set Provides an alternative to classical statistical analysis. –No assumption about the underlying distribution of the data. Deals with outliers. Why study? –Many measures are geometric in nature. –Can be computationally expensive to compute depth.
Examples Half-Space (Tukey, Location) (Tukey 75) Regression Depth (Rousseeuw and Hubert 94) Simplicial Depth (Liu 90) … and many more. 3 Data Points in this Half-plane 2 Data Points in this Half-plane
Desirable Properties of Data Depth Liu (90) / Serfling and Zuo (00) –P1 – Affine Invariance –P2 – Maximality at Center –P3 – Monotonicity Relative to Deepest Point –P4 – Vanishing at Infinity We propose (BRS 04) –P5 – Invariance Under Dimensions Change
Affine Invariance (P1) A – affine transformation
Maximality at Center (P2) p is the center q is any point
Monotonicity Relative to Deepest Point (P3) p is the deepest point q is any point point between p and q
Vanishing at Infinity (P4) q is far from the data cloud
Invariance Under Dimensions Change (P5) Is this an data set?
Simplicial Depth (Liu 90) The simplicial depth of a point p with respect to a probability distribution F in is the probability that a random closed simplex in contains p. where is a closed simplex formed by d+1 random observations from F. The simplicial depth of a point p with respect to a data set in is the fraction of closed simplicies formed by d+1 points of S containing p. where I is the indicator function.
Sample Version of Simplicial Depth The simplicial depth of a point p with respect to a data set in is the fraction of closed simplicies formed by d+1 points of S containing p. Total number of simplicies= =20 ( ) 6363 p is contained in 6 simplicies The depth of p= = __.2.3.4
Properties Is a statistical depth function in the continuous case. (Liu 90) Is affine invariant (P1) and vanishes at infinity (P4) in the sample case. (Serfling and Zuo 00)
Problems in the Sample Case Does not always attain maximality at the center (P2) and does not always have monotonicity relative to the deepest point (P3). (Serfling and Zuo 00) The depth on the boundary of cells is at least the depth in each of the adjacent cells – causes discontinuities. Does not have invariance under dimensions change (P5).
Simplicial Depth (Liu 90) Total number of simplicies = ( ) = A BC D E Averaging number of closed and open simplicies containing a point.3 (BRS 04).7.4 X Y
Revised Definition (BRS 04) The simplicial depth of a point p with respect to a data set in is the average of the fraction of closed simplicies containing p and the fraction of open simplicies containing p, formed by d+1 points of S. Equivalently - the fraction of simplicies with data points as vertices which contain p in their open interior. - the fraction of simplicies with data points as vertices which contain p in their boundary.
Properties of the Revised Definition Reduces to the original definition, for continuous distributions and for points lying in the interior of cells. Keeps ranking order of data points Can be calculated using the existing algorithms, with slight modifications. Fixes Zuo and Serflings counterexamples. The depth on the boundary of two cells is the average of the two adjacent cells. Invariant under dimensions change (P5) for the change from to.
Invariance Under Dimension Change (P5) Degenerate simplicies –Both points C and A (a point between B and C) lie within the open (degenerate) simplex BCD – think of it as a very thin triangle. –Both points B and D are vertices of the (degenerate) simplex BCD. For a point, p, consider the ratio: –For both definitions, the ratio for a position (non-data point) is 2/3. –For Lius definition, the ratio for a data point is not 2/3. –For the BRS definition, the ratio for a data point is 2/3.
Remaining Problems (P2 and P3)
Remaining Problems (Data Points) Data points are still over counted – there can still be discontinuities at data points. However, to fix the depth at data points, more features need to be considered. –Data points are inherently part of simplicies (a point makes a triangle with every other pair of points) and edges are inherently part of simplicies (the two endpoints of an edge make a triangle with every other vertex). –To retain invariance under dimensions change (P5), given a data set in, which lies on a d-flat, then the depth of a point when the data set is evaluated as a d-dimensional data set should be a multiple of the depth when the data set is evaluated as a b-dimensional data set. Neither of the above ideas completely solve the problem and it appears that the best solutions take into account the geometry of the entire data set.
Ongoing Work The current algorithm for finding the median (the deepest point) is O(n 4 ) to walk an arrangement of O(n 2 ) segments. –We can improve this algorithm by comparing simplicial depth and half-space depth. –We are further improving this by considering simplicial depth in the dual. The problems with data points are improved by generalizing this work to higher dimensions. To find the depth at all points, we are using local information to form an approximation for the depth measure.
References G. Aloupis, C. Cortes, F. Gomez, M. Soss, and G. Toussaint. Lower bounds for computing statistical depth. Computational Statistics & Data Analysis, 40(2): , G. Aloupis, S. Langerman, M. Soss, and G. Toussaint. Algorithms for bivariate medians and a Fermat-Torricelli problem for lines. In Proc. 13 th CCCG, pages , M. Burr, E. Rafalin, and D. L. Souvaine. Simplicial depth: An improved definition, analysis, and efficiency for the sample case. Technical report , DIMACS, A. Y. Cheng and M. Ouyang. On algorithms for simplicial depth. In Proc. 13 th CCCG, pages 53-56, J. Gil, W. Steiger, and A. Wigderson. Geometric medians. Discrete Math., 108(1- 3):37-51, Topological, algebraical and combinatorial structures. Frolík's memorial volume. S. Khuller and J. S. B. Mitchell. On a triangle counting problem. Inform. Process. Lett., 33(6): , R. Liu. On a notion of data depth based on random simplices. Ann. of Statist., 18: , Y. Zuo and R. Serfling. General notions of statistical depth function. Ann. Statist., 28(2): , 2000.