Roee Litman, Alexander Bronstein, Michael Bronstein Diffusion-geometric maximally stable component detection in deformable shapes Diffusion-geometric maximally stable component detection in deformable shapes Roee Litman, Alexander Bronstein, Michael Bronstein

MSER In a nutshell… Shape Diffusion Geometry MSER The Feature Approach for Images Deformable Shape Analysis MSER Maximally Stable Extremal Region Shape MSER Diffusion Geometry

Feature Approach in Images Feature based methods are the infrastructure laid in the base of many computer vision algorithms: Content-based image retrieval Video tracking Panorama alignment 3D reconstruction form stereo

Problem formulation Find a semi-local feature detector High repeatability Invariance to isometric deformation Robustness to noise, sampling, etc. Add discriminative descriptor

The “what” Results

Visual Example

Visual Example

(Taken from the TOSCA dataset) More Results (Taken from the TOSCA dataset) - part human, part horse - partial matching - how to match? Horse regions + Human regions

1st, 2nd, 4th, 10th, and 15th matches Region Matching -now let's test it on real data Query 1st, 2nd, 4th, 10th, and 15th matches

3D Human Scans Taken from the SCAPE dataset

Scanned Region Matching Query 1st, 2nd, 4th, 10th, and 15th matches

Volume & surface isometry Volume vs. Surface Original Volume & surface isometry Boundary isometry

Volumetric Shapes Usually shapes are modeled as 2D boundary of a 3D shape. Volumetric shape model better captures "natural" behavior of non-rigid deformations. (Raviv et-al) Diffusion geometry terms can easily be applied to volumes 2D Meshes can be voxelized

Volumetric Regions Taken from the SCAPE dataset

(The “how”) Methodology

Original MSER (Matas et-al) Show some epipolar lines :)

MSER Popular image blob detector Near-linear complexity: High repeatability [Mikolajczyk et al. 05] Robust to affine transformation and illumination changes

MSER – In a nutshell Threshold image at consecutive gray-levels Search regions whose area stay nearly the same through a wide range of thresholds Efficient detection of maximally stable regions requires construction of a component tree

MSER – In a nutshell SKIP?

Algorithm overview

Algorithm overview Represent as weighted graph Component tree Stable component detection Represent as weighted graph Component tree Stable component detection

Algorithm overview Represent as weighted graph Component tree Stable component detection Represent as weighted graph

Image as weighted graph An undirected graph can be created from an image, where: Vertices are pixels Edges by adjacency rule, e.g. 4-neiborhood

Weighting the graph In images Gray-scale as vertex-weight Color as edge-weight [Forssen] In Shapes Curvature (not deformation invariant) Diffusion Geometry

Weighting Option For every point on the shape: Calculate the prob. of a random walk to return to the same point. Similar to Gaussian curvature Intrinsic, i.e. – deformation invariant

Weight example Color-mapped Level-set animation

Diffusion Geometry Analysis of diffusion (random walk) processes Governed by the heat equation Solution is heat distribution at point at time

Heat-Kernel Given Solve using: i.e. - find the “heat-kernel” Initial condition Boundary condition, if these’s a boundary Solve using: i.e. - find the “heat-kernel”

Probabilistic Interpretation The probability density for a transition by random walk of length , from to

Spectral Interpretation How to calculate ? Heat kernel can be calculated directly from eigen-decomposition of the Laplacain By spectral decomposition theorem:

Laplace-Beltrami Eigenfunctions

Deformation Invariance

Computational aspects Shapes are discretized as triangular meshes Can be expressed as undirected graph Heat kernel & eigenfunctions are vectors Discrete Laplace-Beltrami operator Several weight schemes for is usually discrete area elements

Computational aspects In matrix notation Solve eigendecomposition problem

Scale Space The time parameter of the heat kernel spans different scales of transition length is not invariant to shape’s scale Commute-time kernel - scale invariant Probability of a transition by random walk of any length

Auto-diffusivity Special case - The chance of returning to after time Related to Gaussian curvature by Now we can attach scalar value to shapes!

Weight example Color-mapped Level-set animation

Algorithm overview Represent as weighted graph Component tree Stable component detection Represent as weighted graph Component tree Stable component detection

The Component Tree Tree construction is a pre-process of stable region detection Contains level-set hierarchy, i.e. nesting relations. Constructed based on a weighted graph (vertex- or edge-weight) Tree’s nodes are level-sets (of the graph’s cross-sections)

Tree Example

Preliminaries - 1/3 We focus on the undirected graph with the vertex set and edge set , denoted Vertices and are adjacent if The ordered sequence is a path if every consecutive pair is adjacent and are linked A graph is connected if every vertex pair is linked

Preliminaries - 2/3 A graph is called a sub-graph of Given , we take only vertices belonging to an edge A graph is a (connected) component of if it is a maximal connected sub-graph

Preliminaries - 3/3 Vertex-weighted graph - Edge-weighted graph - A cross-section is a sub-graph of a weighted graph, with all weights Level-set is a (connected) component of a cross-section Altitude of a level-set is the maximal weight it contains (can be smaller than ) vertex weight are scalar only edge weight are more general

“Graphic” Example A graph Edge-weighted 7 Cross-Section Two 5 level-sets (with altitude 4) Every level-set has Size (area) Altitude (maximal weight) 4 7 8 9 8 1 4 1

Tree Construction Iterate over vertices by order of weight Create a new component from vertex If vertex is adjacent to existing component(s) add exiting component’s vertices to new one Store component’s area & weight

Tree Construction 1 4 7 8 9 4 7 8 9 8 1 4 1

Algorithm overview Represent as weighted graph Component tree Stable component detection Represent as weighted graph Component tree Stable component detection

Detection Process For every leaf component in the tree: “Climb” the tree to its root, creating the sequence: Calculate component stability Local maxima of the sequence are “Maximally stable components”

The Detils Performance

Benchmarking The Method Method was tested on SHREC 2010 data-set: 3 basic shapes (human, dog & horse) 9 transformations, applied in 5 different strengths 138 shapes in total Scale Original Deformation Holes Noise

Results

Quantitative Results Vertex-wise correspondences were given Regions were projected onto another shape, and overlap ratio was measured Overlap ratio between a region and its projected counterpart is Repeatability is the percent of regions with overlap above a threshold

Repeatability 65% at 0.75

Conclusion Stable region detector for deformable shapes Generic detection framework: Vertex- and edge-weighted graph representation Works on surface and/or volume data Partial matching & retrieval potential Tested quantitatively (on SHREC10)

Thank You Any Questions?