1. 1 Deformation Invariant Shape and Image Matching Polikovsky Senya Advanced Topics in Computer Vision Seminar Faculty of Mathematics and Computer Science Weizmann Institute May 2007

2. 2 Based on… Integral Invariants for Shape Matching Siddharth Manay, Daniel Cremers, Member, Byung-Woo Hong,Anthony J. Yezzi Jr., and Stefano Soatto Deformation Invariant Image Matching Haibin Ling,David W. Jacobs

3. Part I : Invariant Shape Matching Can you guess what it is ? 3

4. 4 Outline Integral Shape Matching Introduction –Basic Definitions, Previous Work –Curvature Integral Invariant (II) – Relation of Local Area II to Curvature – Shape Matching and Distance – Multi-scale Shape Matching Implementation and Experimental Results

5. 5 Applications of shape matching Airport security : Industry quality control : Medical images analyses :

6. 6 Ultimate Goal Compare objects represented as closed planar contours.

7. 7 Basic Definition Object : –Closed planar contour. –No self-intersections. Shape : –Equivalence class of objects. Obtained under the action of a finite-dimensional group : Euclidean, similarity, affine, projective group.

8. 8 Basic Definition (cont.) In Addition: - articulated - occluded - jagged (not obtained with standard additive, zero mean) Two objects have the same shape if and only if one can be generated by transformation group actions on other shape.

9. 9 Summary Goal Define a distance so that shapes that vary by Euclidean transformations have zero distance. Shapes vary by scaling, articulation, occlusion have small distance. Resistant to : “small deformations” “high-frequency noise” “localized changes”

10. 10 Summary Goal (cont.) The babies should have low distance to each other, but high distance to other classes of shapes. Low distanceHigh distance

11. 11 Previous Work Statistical approach using moments. [35],[27] High-order moments sensitive to noise. Normalized Fourier descriptors. [93],[52],[2] High order Fourier coefficients are not stable with respect to noise. Local neighborhoods using Wavelet transform. [83],[34]

12. Previous Work (Cont.) 12 Global radial histogram of the relative coordinates of the rest of the shape at each point. [6] Differential invariants (Curvature) [44], [36], [17], [58], [76], [30], [48], [64], [91], [85], [37]

13. 13 Curvature Intuitively, curvature is the amount by which a geometric object deviates from being flat. Less general y = f(x ) Plane curve c(t) = (x(t),y(t))

14. Curvature can be also be geometrical understood as in terms of osculating (kiss) circle and radius curvature. 14 Curvature (cont.) Circle of radius r has curvature 1/r everywhere. Straight line r = ∞ has curvature 1/ ∞ =0 everywhere Features: –Invariant to rotation, reflections of the original curve. –NOT invariant to scaling.

15. 15 Outline Integral Shape Matching Introduction –Basic Definitions, Previous Work –Curvature Integral Invariant (II) – Relation of Local Area II to Curvature – Shape Matching and Distance – Multi-scale Shape Matching Implementation and Experimental Results

16. 16 Closed planar contour. C:S 1 → R 2. ds - infinitesimal arclength. G - transformation group acting on R 2. dx - area %on R 2. μ curve C corresponding measure dμ(x). Integral Invariant

17. 17 Integral Invariant Function I: R 2 → R G-invariant if satisfies: I(.) is associates to each point on contour a real number.

18. 18 Step 1: Curvature Curvature κ(C) of curve C is G-invariant.

19. 19 General notion of Integral Invariant Function I C (p) : R 2 → R is and integral G-invariant Kernel k : R 2 X R 2 → R. K( ●, ● ) : p don’t necessarily lie on the curve C. p

20. 20 Step 2: Distance integral invariant Unlike curvature distance invariant is R +. (Euclidean distance is always nonnegative) For global descriptor, local change of a shape affects the values of the distance integral invariant for the entire shape.

21. 21 Step 2: Distance integral invariant

22. 22 Step 3: “Shape Context” Preserves locality can be obtained by weighting the integral with a kernel q(p, x). Local radial histogram.

23. 23 Step 3: “Shape Context” (cont.) NOT discriminative, same value for different geometric features.

24. r – radius. p - center of the ball. C - interior of the region bounded by C. 24 Finale Step: Local Area Integral Invar. Define a ball B r (p), B r : R 2 X R 2 {0,1} ; ( R + )

25. 25 Local Area Integral Invar.(cont.) r r Naturally forms a multi-scale invariant.

26. 26 Relation of Local AII to Curvature r C r R θ C Assume that C is smooth curve, because of the curvature.

27. 27 Outline Integral Shape Matching Introduction –Basic Definitions, Previous Work –Curvature Integral Invariant (II) – Relation of Local Area II to Curvature – Shape Matching and Distance – Multi-scale Shape Matching Implementation and Experimental Results

28. 28 Shape Matching and Distance Shape distance is a scalar value that quantifies similarity of the two contours. D(C 1,C 2 ). Basing on group invariant, integral invariant : –invariant to G group action. –robust to noise and local deformations Corresponding points. C 1 = C2=C2= Local Are Integral Invariant : I 1, I 2,

29. 29 Corresponding points Disparity function d(s). Reparameterizes C 1,I 1 and C 2,I 2. optimal point correspondence ( ~ denotes correspondence)

30. 30 Energy Functional E(...,d) E 1 - measures the similarity of two curves. E 2 - elastic energy. α - control parameter α > 0. If d(s) = 0, direct match. If d’(s) = 0, circular “shifts”. Other d(s), “stretch”, “shrink”.

31. 31 Control Parameter α small α : large α : d*(s)

32. 32 Multi-scale Shape Matching Trace of local extrema across scales Curvature Scale-space Integral Invariant Scale-space Curvature scale-space is derived from Gaussian smoothing.

33. 33 Multi-scale Shape Matching (cont.) Matching shapes of different sizes. R R’ Normalized kernel radius : r / R.

34. 34 Multi-scale Shape Matching (cont.) Control parameter α. Size of the kernel width r. Correspondences between two signals influenced by: fine scale intermediate scale coarse scale

35. 35 Outline Integral Shape Matching Introduction –Basic Definitions, Previous Work –Curvature Integral Invariant (II) – Relation of Local Area II to Curvature – Shape Matching and Distance – Multi-scale Shape Matching Implementation and Experimental Results

36. 36 Implementation C1C1 C2C2 C 2 [j] C 2 [j+1] C 1 [i] C 1 [i+1] v[i, j] v[i, j + 1] v[i + 1, j] v[I + 1, j + 1] Each point in each curve must have at least one corresponding point in the other curve. v[i, j] v[I + 1, j + 1] v[i +1, j] v[i, j+1] e e e e = v(i,j) → v(k,l) Minimization of the energy functional E.

37. 37 Implementation (cont.) Minimization of the energy functional E is equivalent to finding a shortest path that gives a minimum weight. w(p) ← E(I 1,I 2,d) Graph used to compute the correspondence for two curves with M = N = 5. nodes = MN edges = 3MN v[i, j] v[I + 1, j + 1] v[i +1, j] v[i, j+1] e e e M N 0 0 1 1 C1C1 C2C2

38. 38 Implementation (cont.) In previous implementations we choose start point in C 1 and run all over C 2. Alternately, observing strong features, in the invariant space.

39. 39 Experimental Results The gray levels indicate the dissimilarity between points lighter shade indicates higher dissimilarity. dissimilarity higher dissimilarity

40. 40 Experimental Results (cont.) “On Aligning Curves” [76] 100 sample on contours r = 15 α = 0.1

41. 41 Experimental Results (cont.) Int.Inv. Curvature Histograms of shape distance between Shape 24 and 1,000 perturbations of Shape 20 with noise at variance = 2.5. Int.Inv. Curvature

42. 42 Experimental Results (cont.) Noisy shapes (across top) and original shapes (along left side) vie differential invariant dissimilarity higher dissimilarity

43. 43 Experimental Results (cont.) Noisy shapes (across top) and original shapes (along left side) via integral invariant dissimilarity higher dissimilarity

44. 44 Summary Curvature. Four Steps : –Curvature, sensitivity to noise. –Global descriptor, local change. –Local radial histogram, NOT discriminative. –Local Area Integral Invar. Shape Matching,Multi-scale Shape Matching. Implementation. Results. M N 0 0 1 1 C1C1 C2C2

45. Part II : Deformation Invariant Image Matching 45

46. Outline Introduction Deformation Invariant Framework Experiments Summary

47. General Deformation One-to-one, continuous mapping. Intensity values are deformation invariant. –(their positions may change) Affine model for lighting change. (Out of the scope)

48. Solution A deformation invariant framework –Embed images as surfaces in 3D –Geodesic distance is made deformation invariant by adjusting an embedding parameter –Build deformation invariant descriptors using geodesic distances

49. Related Work Embedding and geodesics –Beltrami framework [Sochen&etal98] –Bending invariant [Elad&Kimmel03] –Articulation invariant [Ling&Jacobs05] Histogram-based descriptors –Shape context [Belongie&etal02] –SIFT [Lowe04] –Spin Image [Lazebnik&etal05, Johnson&Hebert99] Invariant descriptors –Scale invariant descriptors [Lindeberg98, Lowe04] –Affine invariant [Mikolajczyk&Schmid04, Kadir04, Petrou&Kadyrov04] –MSER [Matas&etal02]

50. Outline Introduction Deformation Invariant Framework - Intuition through 1D images - 2D images Experiments Summary

51. 1D Image Embedding 1D Image I(x): EMBEDDING I(x)  ( (1-α)x, αI ) αIαI (1-α)x Aspect weight α : measures the importance of the intensity 2D Surface :

52. Geodesic Distance αIαI (1-α)x p q g(p,q) Length of the shortest path along surface

53. Geodesic Distance and α I1I1 I2I2 Geodesic distance becomes deformation invariant for α close to 1

54. Image Embedding & Curve Lengths Depends only on intensity I  Deformation Invariant Image I: Embedded Surface σ : Curve γ on σ: Length of γ: Take limit

55. Geodesic distances for real images 55 Automatically adapts to deformation. Almost like Euclidean distances Interest point p 0 = (x0, y0). Compute Geodesic distances from p 0 to all other points on embedded surface σ(I; α). α = 0 α = 0.98

56. Geodesic Distance for 2D Images Computation –Geodesic level curves –Fast marching [Sethian96] is the marching speed Geodesic distance –Shortest path –Deformation invariant F T is the geodesic distance T=1 T=2 T=3 T=4 p q

57. Deformation Invariant Sampling Geodesic Sampling 1.Fast marching: get geodesic level curves with sampling interval Δ 2.Sampling along level curves with Δ p sparse dense Δ Δ Δ Δ Δ

58. Geodesic-Intensity Histogram(GIH) Divide 2D intensity-geodesic distance space into K×M bins. K - number of intensity intervals. M - number of geodesic distance intervals. Insert all points in Pp into Hp. Normalize each column of Hp Normalize the whole Hp. Deformation Invariant Descriptor p q p q geodesic distance M Intensity K geodesic distance M Intensity K

59. Real Example p q

60. Invariant vs. Descriminative

61. Deformation Invariant Framework Image Embedding ( α close to 1) I(x,y) → σ(I, α) Deformation Invariant Sampling Geodesic Sampling Build Deformation Invariant Descriptors (GIH)

62. Practical Issues Interest-Point –No special interest-point is required. Automatically locates the support region by Geodesic Sampling. But: – Points on constant region indistinguishable. – Corners may vary due to sampling. Extreme point, local intensity extremum, is more reliable and effective.

63. Outline Introduction Deformation Invariant Framework - Intuition through 1D images - 2D images Experiments -Interest-point matching Summary

64. Data Sets Synthetic Deformation & Lighting Change (8 pairs) Real Deformation (3 pairs)

65. Interest-Points Interest-point Matching Harris-affine points [Mikolajczyk&Schmid04] * Affine invariant support regions Not required by GIH 200 points per image Ground-truth labeling Automatically for synthetic image pairs Manually for real image pairs Correct match, three pixel distance.

66. Descriptors & Performance Evaluation Descriptors GIH compared with following descriptors: Steerable filter [Freeman&Adelson91], SIFT [Lowe04], Moments [VanGool&etal96], Complex filter [Schaffalitzky&Zisserman02], Spin Image [Lazebnik&etal05] α = 0.98 Performance Evaluation Receiver Operating Characteristics (ROC) curve: detection rate among top N matches. Detection rate:

67. Synthetic Image Pairs N – top matches.

68. Real Image Pairs N – top matches.

69. Outline Introduction Deformation Invariant Framework - Intuition through 1D images - 2D images Experiments -Interest-point matching Summary

70. 1D Image Intuition. Geodesic Distance and α. Geodesic Sampling. Performance Evaluation. p Δ Δ Δ Δ Δ

71. Thank You!