Non-Euclidean Example: The Unit Sphere

Differential Geometry Is the mathematical theory which… …works with small ‘patches’ looking Euclidean… …and do calculus over these patches.

Manifolds Open Sets Coordinate neighbourhood Compatible neighbourhoods Talk about analytic manifold

Tangent Space Tangent Vectors Tangent Space, Inner Product along D Norm: depends on varies smoothly

Geodesics and Metrics The shortest path between two points X, Y is the geodesic in the manifold. The length of the geodesic is the distance between the two points

Exponential and Logarithm Maps : Maps tangents to the manifold : Maps points on the manifold to Both maps are locally defined only

Gradient In euclidean space: direction of fastest increase On the manifold: tangent of fastest increase is a real valued function on the manifold The gradient at X is is unique satisfying directional derivative along D

Vector Space vs. Manifold Space X the “origin”, Y an other point X. Pennec, P. Fillard and N. Ayache , “A Riemannian Framework for Tensor Computing,” International. Journal of Computer Vision., 66(1), 41–66, 2006.

Sets of matrices, general linear group GL(n,R) Lie Groups Sets of matrices, general linear group GL(n,R) form a group under matrix multiplication are the simplest Riemannian manifolds Examples Rigid body transformations SE(n) (n+1)x(n+1) Rotations SO(n) nxn Affine motions A(n) (n+1)x(n+1) R t 0T 1 W. Rossman, “Lie Groups: An Introduction through Linear Groups,” Oxford University Press, 2003.

Therefore the representation is not unique! Grassmann Manifolds, . Each point on the Grassmann manifold, , represents a dimensional subspace of . Numerically, represented by an orthonormal basis (does not matter which one) matrix such that Therefore the representation is not unique! computation should account for this A. Edelman, T. A. Arias and S. T. Smith, “The Geometry of Algorithms with Orthogonality Constraints,” SIAM Journal on Matrix Analysis and Applications, 20(2), 303–353, 1998.

Equivalent to SO(3)xSO(3) Essential Manifold calibrated cameras Set of matrices with two equal and one zero singular value let the two equal singular values be 1 Equivalent to SO(3)xSO(3) two-time covering of the essential manifold Correction by Dubbelman(2012). Subbarao&Meer(2009) had two solutions from eight possible ones, and one still had to be eliminated S. Soatto, R. Frezza and P. Perona , “Recursive Estimation on the Essential Manifold,” 3rd Europan Conference on Computer Vision, Stockholm, Sweden, May 1994, vol.II, p.61-72.

Twisted pairs gives the same essential matrix Four different camera geometries, but the same essential matrix.

Symmetric Positive Define (SPD) Manifold contains symmetric positive definite matrices. e.g. diffusion tensor MRI has two different metrics Affine Invariant metric (precise) Log-Euclidean Metric (this is used in comp.vision) practically similar to the affine invariant metric computationally easier to work with V. Arsigny , P. Fillard, X. Pennec and N. Ayache , “Geometric Means in a Novel Vector Space Structure on Symmetric Positive-Definite Matrices,” SIAM Journal of Matrix Analysis and Applications, 29(1), 328–347, 2006.

Mean Shift for Euclidean Spaces The kernel density estimate Mean shift as normalized gradient of where The iteration

Gradient ascent on the kernel density on the closest mode. Original Mean Shift Gradient ascent on the kernel density on the closest mode. Equivalent to expectation-maximization for Gaussian kernels Mean shift is a nonparametric clustering D. Comaniciu and P. Meer , “Mean Shift: A Robust Approach Towards Feature Space Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, 603–619, 2002. D. Comaniciu, V. Ramesh and P. Meer , “Kernel-based Object Tracking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.25, 564–577, 2003.

Mean Shift for Manifolds The kernel density estimate on the manifold Mean shift as normalized gradient of , tangent space The iteration

Mean Shift for Riemannian Manifolds Map locally the points to tangent space Compute the weighted average of the tangent vectors and obtain the mean shift vector Map the mean shift vector back to the manifold

Properties for Nonlinear Mean Shift Still the gradient ascent on the kernel density Nonlinear mean shift is convergent but depend on an upper limit on allowed bandwidth in the tangent space! nonlinear mean shift equivalent to EM for Gaussians kernels. Valid for the groups mentioned. All significant modes may come out in the manifold.

Clustering by nonlinear mean shift Motion Segmentation Hypothesis Generation matrix Lie groups Randomly pick N elemental subset of corresponding points. May have outlier elemental subsets too. Generate N parameter hypotheses R, t Clustering by nonlinear mean shift the clustering needs h(-s) Returns Number of dominant modes Positions of dominant modes

2D Affine Motion A(2) matrix 3x3 Total of 80 matches. Affine has 6 parameters. Three point pairs per elemental subset. 500 clustering hypotheses.

Camera Pose Estimation SE(3) 4x4 matrix Calibrated camera. Three point clouds, triangulated in 3D have different coordinate systems. With all three scenes present, segmentation in 2D can be used to establish 2D to 3D correspondences for the entire scene. 3D poses are estimated with SE(3) and h=0.1.

Results with all three scenes together M1 books on right M2 books on left M3 box in middle

Factorization G10,3 uncalibrated affine camera Grassmann. 5 frames n=2x5=10. planar move each object k=3. e.subset 3 points in all 5 frames. 257 corners. 1000 hypotheses. h = 0.07

Essential Matrix SO(3)xSO(3) Two motions. 100 point matches. 1000 hypotheses. h = 0.001 Second solution comes from rotation +-p around the y-axis, which maps into the same tangent vector.

Discontinuity Preserving Filtering An image is a mapping from a lattice in to data lying on a manifold Filtering: Run mean shift in the space 2D hs Mean shift iterations update the spatial and parameter values. The iteration from converges to set in the filtered image

Chromatic Noise Filtering G3,1 Chromaticity affects direction of the color vector but not the intensity. s = 0.2 original mean shift hs=11 hp=7 chromatic hs=11 hp=0.7

Chromatic Noise Filtering original mean shift hs=11 hp=10.5 chromatic hs=11 hp=0.3

Diffusion tensor Image R3xSym+3 Images encoded with 3x3 Symmetric Positive Definite (SPD) matrices at each pixel. Smoothing the DTI improves the result.

Group theory starts to be used as part of a deep learning method. Example: Lohit,Turaga: Learning Invariant Riemannian Geometric Representations Using Deep Nets. https://arxiv.org/pdf/1708.09 Grassmannian manifold multi-class classification on a unit hypersphere Exmp: Face to illumination subspace 28x28 images G784,d Orthonornal matrix U:784xd The grassmannian manifold is invariant to orthogonal transformations, like illuminations, which is in a low dim. subspace. The 90% of the variances are in the first d=5 principal components of a database. 250(200+50) 3D random faces. 33 illuminations from Yale database B, which had a total of 64 illuminations. Neural network has 3 convolution layers and 2 full connected. || Frechet (geometric) mean in the tangent space for the ground-truth subspace of the training set. || All the results are projected in the manifold.

gives RQ*, a new point of the manifold. d=5 d=4 T DG the mean subspace distance of principal angles obtain from Uavg Utest Returns the subspace of illumination with a point of the manifold R which spans the same subspace as the columns of U. The rotation Q* = min ||Uavg – RQ||F gives RQ*, a new point of the manifold. orthogonal Procrustes problem