1. Eduard Serradell Domingo July 28 th, 2011 Simultaneous Point Matching and Recovery of Rigid and Nonrigid Shapes Thesis director Francesc Moreno Noguer Tutor Alberto Sanfeliu Cortés T HESIS P ROPOSAL

2. Objective Simultaneously solve the correspondence problem and recover rigid and nonrigid shapes. Given two point clouds extracted from different views of the same object, the objective is to simultaneously solve for point correspondence and recover the mapping between the two rigid or nonrigid model representations.

3. Motivation  Given {u m } from a model point set and {v t } from a target point set, find: model point set target point set

4. Motivation CORRESPONDENCES  Given {u m } from a model point set and {v t } from a target point set, find: umum vtvt

5. Motivation CORRESPONDENCES TRANSFORM ESTIMATION H est  Given {u m } from a model point set and {v t } from a target point set, find: umum vtvt

6. Motivation CORRESPONDENCES TRANSFORM ESTIMATION H est  Given {u m } from a model point set and {v t } from a target point set, find: umum vtvt SIMULTANEOUSLY

7. Motivation  Common problems: correspondences that do not fit the model umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

8. Motivation  Common problems: partial matching parts of the scene are occluded 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D umum vtvt

9. Motivation  Common problems: points that do not belong to the model and hinder the recognition umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

10. Motivation  Common problems: error in the position of points umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

11. Motivation  Common problems: regular structures are indistinguishable algorithms fall into local minima umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

12. Motivation  Common problems: umum vtvt model shapes can undergo rigid or nonrigid deformations 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

13. Motivation  Common problems: model shapes can undergo rigid or nonrigid deformations umum vtvt 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

14. Motivation  Common problems: umum vtvt the transform can be embedded in 2D or 3D or projective as in monocular view case (2D-3D) Coordinate System 1 Coordinate System 2 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

15. Motivation  Common problems: umum vtvt u m = R · v t + t Coordinate System 1 Coordinate System 2 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

16. Motivation  Common problems: World Coordinate System Camera Coordinate System the transform can be embedded in 2D or 3D or projective as in monocular view case (2D-3D) 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

17. Motivation  Common problems: World Coordinate System Camera Coordinate System x cam = A [R| t] x world 1.Outliers 2.Occlusions 3.Cluttered background 4.Observation noise 5.Repetitive patterns 6.Rigid / Nonrigid model 7.2D/3D or 2D-2D / 3D-3D

18. State-of-Art 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Rigid 2D-2D / 3D-3DRigid 2D-3D Nonrigid 2D-2D / 3D-3D Nonrigid 2D-3D

19. Nonrigid 2D-2D / 3D-3D Rigid 2D-3D 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model State-of-Art Rigid 2D-2D / 3D-3D

20. State-of-Art RIGID 2D-2D / 3D-3D MATCHING 1970 1980 1990 2000 2010 [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography RANSAC (global solution / high complexity)

21. State-of-Art RIGID 2D-2D / 3D-3D MATCHING [1992] Besl & McKay, A Method for Registration of 3D Shapes 1970 1980 1990 2000 2010 [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography ICP : Iterative Closest Point (requires good initialization)

22. State-of-Art RIGID 2D-2D / 3D-3D MATCHING 1970 1980 1990 2000 2010 [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography [1992] Besl & McKay, A Method for Registration of 3D Shapes [2005] Chum & Matas. Matching With PROSAC - Progressive Sample Consensus PROSAC: RANSAC + Appearance

23. State-of-Art RIGID 2D-2D / 3D-3D MATCHING [1992] Besl & McKay, A Method for Registration of 3D Shapes 1970 1980 1990 2000 2010 [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2005] Chum & Matas. Matching With PROSAC - Progressive Sample Consensus Unsolved when: Weak detection, -outliers, -occlusions, -image noise, -repetitive patterns, -highly textured scenes, -oblique angles -… PROSAC complexity similar to RANSAC (too high)

24. Rigid 2D-2D / 3D-3D Nonrigid 2D-3D Nonrigid 2D-2D / 3D-3D 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model State-of-Art Rigid 2D-3D

25. State-of-Art Known correspondences : Perspective-n-Point (PnP) (old problem) RIGID 2D-3D MATCHING 1970 1980 1990 2000 2010 [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem

26. State-of-Art RIGID 2D-3D MATCHING 1970 1980 1990 2000 2010 [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography RANSAC: Random Sampled Consensus & DLT: Direct Linear Transform (high complexity)

27. State-of-Art RIGID 2D-3D MATCHING 1970 1980 1990 2000 2010 [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2002] David et al,SoftPOSIT: Simultaneous Pose and Correspondence Determination SoftPOSIT: Unknown correspondences

28. State-of-Art RIGID 2D-3D MATCHING 1970 1980 1990 2000 2010 [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2002] David et al,SoftPOSIT: Simultaneous Pose and Correspondence Determination [2008] Moreno-Noguer et al, Pose Priors for Simultaneously Solving Alignment and Correspondence. Blind PnP: Unknown correspondences Pose Priors (geometrical consistency) Kalman Filter to propagate pose uncertainty

29. State-of-Art RIGID 2D-3D MATCHING 1970 1980 1990 2000 2010 [2009] Moreno-Noguer et al, EPnP: An Accurate O(n) Solution to the PnP Problem [1981] Fischler & Bolles. Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography [2002] David et al,SoftPOSIT: Simultaneous Pose and Correspondence Determination [2008] Moreno-Noguer et al, Pose Priors for Simultaneously Solving Alignment and Correspondence. Modeling the uncertainty : Kalman Filter linearizes the uncertainty model Work with Bayesian non-parametric models (Gaussian Processes)

30. State-of-Art 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D

31. State-of-Art 1970 1980 1990 2000 2010 NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence Soft-assign + Thin-plate splines (requires good initialization) (smooth deformations) Soft-assign (requires good initialization)

32. State-of-Art 1970 1980 1990 2000 2010 [2003] Hannel et al.. An Extension of the ICP Algorithm for Modeling Nonrigid Objects with Mobile Robots NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [2008] Li et al, Global Correspondence Optimization for Non-Rigid Registration of Depth Scans [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence Nonrigid ICP Variants (Require a good initialization)

33. State-of-Art 1970 1980 1990 2000 2010 [2003] Hannel et al.. An Extension of the ICP Algorithm for Modeling Nonrigid Objects with Mobile Robots NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [2008] Li et al, Global Correspondence Optimization for Non-Rigid Registration of Depth Scans [2010] Deng et al, Retinal Fundus Image Registration via Vascular Structure Graph Matching [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence [2002] Belongui, Shape matching and Object Recognition Using Shape Contexts Shape appearance + Thin-plate splines (Smooth deformations) Graph Matching + Thin-plate splines (Smooth deformations)

34. State-of-Art 1970 1980 1990 2000 2010 [2003] Hannel et al.. An Extension of the ICP Algorithm for Modeling Nonrigid Objects with Mobile Robots NONRIGID 2D-2D / 3D-3D MATCHING [2003] Chui & Rangarajan. A New Point Matching Algorithm for Non-Rigid Registration [2008] Li et al, Global Correspondence Optimization for Non-Rigid Registration of Depth Scans [2010] Myronenko & Song, Point-Set Registration: Coherent Point Drift [2010] Deng et al, Retinal Fundus Image Registration via Vascular Structure Graph Matching [1998] Gold et al.. New Algorithms for 2D and 3D Point Matching: Point Estimation and Correspondence [2002] Belongui, Shape matching and Object Recognition Using Shape Contexts Unsolved problems Harsh deformations (Gaussian Processes vs TPS) Avoid local minima (Global Search vs ICP et al.)

35. Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D State-of-Art 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D

36. State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures 1970 1980 1990 2000 2010 1970 1980 1990 2000 2010

37. State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures 1970 1980 1990 2000 2010 1970 1980 1990 2000 2010 [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion Discriminative methods: Database learning & Nearest Neigbour Selection

38. State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures 1970 1980 1990 2000 2010 1970 1980 1990 2000 2010 [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion Generative methods: PCA model of the surface

39. State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures 1970 1980 1990 2000 2010 1970 1980 1990 2000 2010 [2009] Groher, Deformable 2D-3D Registration of Vascular Structures in a One View Scenario [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion Thin-plate Splines (Medical Imaging!)

40. State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures 1970 1980 1990 2000 2010 1970 1980 1990 2000 2010 [2010] Salzmann & Urtasun, Combining Discriminative and Generative Methods for 3D Deformable Surface and Articulated Pose Reconstruction [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion [2009] Groher, Deformable 2D-3D Registration of Vascular Structures in a One View Scenario Combining Discriminative & Generative methods

41. State-of-Art Deformable surfaces NONRIGID 2D-3D MATCHING Articulated structures 1970 1980 1990 2000 2010 1970 1980 1990 2000 2010 [2010] Salzmann & Urtasun, Combining Discriminative and Generative Methods for 3D Deformable Surface and Articulated Pose Reconstruction [2010] Sanchez et al., Simultaneous Pose, Correspondence and Non-Rigid Shape [2007] Salzmann et al., Surface Deformation Models for Non-Rigid 3D Shape Recovery [2003] Shakhnarovich et al., Fast Pose Estimation with Parameter Sensitive Hashing [2006] Sigal & Black, Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion [2009] Groher, Deformable 2D-3D Registration of Vascular Structures in a One View Scenario Potential improvements Better parameterization for articulated structures Harsh deformations (Gaussian Processes vs TPS)

42. Nonrigid 2D-3D Nonrigid 2D-2D / 3D-3D 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Contributions Rigid 2D-3D Rigid 2D-2D / 3D-3D

43. Contributions Rigid registration Homography (2D-to-2D) estimation

44. Rigid registration  Feature Point Detectors assigns multiple correspondences PROSAC picks just the one with better similarity score !!! Contributions Homography (2D-to-2D) estimation

45. Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point Contributions Homography (2D-to-2D) estimation

46. Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach Contributions Homography (2D-to-2D) estimation

47. Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach Contributions Homography (2D-to-2D) estimation

48. Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach Contributions Homography (2D-to-2D) estimation

49. Rigid registration  Feature Point Detectors assigns multiple correspondences  Using Kalman Filter we can propagate geometry priors, thus constraining the search regions for each feature point  Iterative approach & backtracking when necessary Contributions Homography (2D-to-2D) estimation

50. Contributions Rigid registration  Some results : PROSACBlind Homography

51. Contributions Rigid registration  E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and F. Moreno-Noguer: Combining Geometric and Appearance Priors for Robust Homography Estimation. In ECCV 2010 Homography (2D-to-2D) estimation

52. Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D Contributions 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D

53. Contributions Nonrigid registration  Nonrigid 2D-to-3D registration  Project in collaboration with  New parameterization for articulated models 2D X-ray ImageCT 3D Volume

54. Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure 2D X-ray Image CT 3D Volume Camera (known)

55. Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure  Generative model: Probabilistic PCA 2D features synthetic samples Camera (known)

56. Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure  Generative model: Probabilistic PCA 2D features generative model Camera (known)

57. Contributions Nonrigid registration  Recursive parameterization of the nodes of the articulated structure  Generative model: Probabilistic PCA  Iterative update 1.- Kalman Filter model projection 2.- Assign correspondences 2D features generative model Camera (known)

58. Contributions Nonrigid registration  E. Serradell, A. Romero, R. Leta, C. Gatta and F. Moreno-Noguer: Simultaneous Correspondence and Non-Rigid 3D Reconstruction of the Coronary Tree from Single X-ray Images. In ICCV 2011 2D X-ray ImageCT 3D Volume shape prior recovered model

59. Contributions 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Nonrigid 2D-3D Rigid 2D-2D / 3D-3D Rigid 2D-3D Nonrigid 2D-2D / 3D-3D

60. Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration Optical Microscope Image Stack Electron Microscope Image Stack 10 -6 m10 -7 m Partial matching

61. Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration  Extract neuronal tree  Graph Matching Optical Microscope Image Stack Electron Microscope Image Stack

62. Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration  Extract neuronal tree  Graph Matching  Two step process 1.- Affine transform (Kalman Filter approach) Optical Microscope Image Stack Electron Microscope Image Stack y = A x + b

63. Contributions Nonrigid registration  Nonrigid 2D-to-2D or 3D-to-3D registration  Extract neuronal tree  Graph Matching  Two step process 1.- Affine transform (Kalman Filter approach) 2.- Nonrigid transform (Gaussian Processes for regression) Optical Microscope Image Stack Electron Microscope Image Stack y = A x + b + f(x)

64. Contributions Nonrigid registration  Some results original graphs affine transform nonlinear refining

65. Contributions Nonrigid registration  Some results original graphs affine transform nonlinear refining affine transform nonlinear refining

66. Contributions Nonrigid registration  E. Serradell, J. Kybic, F. Moreno-Noguer and P. Fua: Robust Elastic 2D/3D Geometric Graph Matching, submitted to SPIE Medical Imaging Optical Microscope Image Stack Electron Microscope Image Stack

67. Contributions 2D-3D2D-2D or 3D-3D Rigid Model Nonrigid Model Rigid 2D-2D / 3D-3DRigid 2D-3D Nonrigid 2D-2D / 3D-3D Nonrigid 2D-3D

68. Global solution to point registration  Valid for 2D-2D,3D-3D,2D-3D / rigid and nonrigid models  Using Gaussian Processes … some preliminary results: Contributions initial shape recovered shape

69. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis

70. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis DoneOn-goingTo Do

71. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Research Stay at CVLAB (EPFL) DoneOn-goingTo Do

72. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Project in collaboration with CVC (UAB), MAiA (UB) and Hospital Sant Pau DoneOn-goingTo Do

73. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Research Stay at CVLAB (EPFL) DoneOn-goingTo Do

74. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis Research Stay at CVLAB (EPFL) DoneOn-goingTo Do

75. 1.Master ARV Task Planning 2.Simultaneous Correspondence & Robust Estimation 3.Nonrigid Model Reconstruction from Single Images 4.Elastic Graph Matching 5.Write Thesis DoneOn-goingTo Do

76. Published papers  E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and F. Moreno-Noguer: Combining Geometric and Appearance Priors for Robust Homography Estimation. In ECCV 2010*  E. Serradell, A. Romero, R. Leta, C. Gatta and F. Moreno-Noguer: Simultaneous Correspondence and Non-Rigid 3D Reconstruction of the Coronary Tree from Single X-ray Images. In ICCV 2011* * ICCV, ECCV acceptance rate < 30% Submitted papers  E. Serradell, J. Kybic, F. Moreno-Noguer and P. Fua: Robust Elastic 2D/3D Geometric Graph Matching, submitted to SPIE Medical Imaging Summary of achievements

77. THANKS!