1. Y. Moses 11 Combining Photometric and Geometric Constraints Yael Moses IDC, Herzliya Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

2. Y. Moses 22 Recover the 3D shape of a general smooth surface from a set of calibrated images Problem 1:

3. Y. Moses 33 Problem 2: Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

4. Y. Moses 44 Shape Recovery  Geometry: Stereo  Photometry:  Shape from shading  Photometric stereo Main problems: Calibrations and Correspondence

5. Y. Moses 55 3D Shape Recovery Photometry:  Shape from shading  Photometric stereo Geometry:  Stereo  Structure from motion

6. Y. Moses 66 Geometric Stereo  2 different images  Known camera parameters  Known correspondence + +

7. Y. Moses 77 Photometric Stereo  3D shape recovery: surface normals from two or more images taken from the same viewpoint

8. Y. Moses 88 Photometric Stereo Solution: Three images Matrix notation

9. Y. Moses 99 Photometric Stereo  3D shape recovery (surface normals) Two or more images taken from the same viewpoint Main Limitation: Correspondence is obtained by a fixed viewpoint

10. Y. Moses  10 Overview  Combining photometric and geometric stereo:  Symmetric surface, single image  Non symmetric: 3 images  Mono-Geometric stereo  Mono-Photometric stereo  Experimental results.

11. Y. Moses  11 The input  Smooth featureless surface  Taken under different viewpoints  Illuminated by different light sources The Problem: Recover the 3D shape from a set of calibrated images

12. Y. Moses  12 Assumptions  Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …) * *  n n  Three or more images  Perspective projection

13. Y. Moses  13 Our method Combines photometric and geometric stereo We make use of:  Given Correspondence:  Can compute a normal  Can compute the 3D point

14. Y. Moses  14 Basic Method Given Correspondence

15. Y. Moses  15 First Order Surface Approximation

16. Y. Moses  16 First Order Surface Approximation

17. Y. Moses  17 First Order Surface Approximation P(  ) = (1 -  )O 1 +  P , N (P(  ) - P) = 0

18. Y. Moses  18 First Order Surface Approximation

19. Y. Moses  19 New Correspondence

20. Y. Moses  20 New Surface Approximation

21. Y. Moses  21 Dense Correspondence

22. Y. Moses  22 Basic Propagation

23. Y. Moses  23 Basic Propagation

24. Y. Moses  24 Basic method: First Order  Given correspondence p i and L  P and n  Given P and n  T  Given P, T and M i  a new correspondence q i

25. Y. Moses  25 Extensions  Using more than three images  Propagation:  Using multi-neighbours  Smart propagation  Second error approximation  Error correction:  Based on local continuity  Other assumptions on the surface

26. Y. Moses  26 Multi-neighbors Propagation

27. Y. Moses  27 Smart Propagation

28. Y. Moses  28 Second Order: a Sphere P()P() N+N  N NN P (P-P(  ))(N+N  )=0

29. Y. Moses  29 Second Order Approximation

30. Y. Moses  30 Second Order Approximation

31. Y. Moses  31 Using more than three images  Reduce noise of the photometric stereo  Avoid shadowed pixels  Detect “bad pixels”  Noise  Shadows  Violation of assumptions on the surface

32. Y. Moses  32 Smart Propagation

33. Y. Moses  33 Error correction The compatibility of the local 3D shape can be used to correct errors of:  Correspondence  Camera parameters  Illumination parameters

34. Y. Moses  34 Score  Continuity:  Shape  Normals  Albedo  The consistency of 3D points locations and the computed normals:  General case: full triangulation  Local constraints

35. Y. Moses  35 Extensions  Using more than three images  Propagation:  Using multi-neighbours  Smart propagation  Second error approximation  Error correction:  Based on local continuity  Other assumptions on the surface

36. Y. Moses  36 Real Images  Camera calibration  Light calibration  Direction  Intensity  Ambient

37. Y. Moses  37 Error correction + multi-neighbor 5 Images

38. Y. Moses  38 5pp 3pp 3nn 5nn 5pn

39. Y. Moses  39

40. Y. Moses  40

41. Y. Moses  41

42. Y. Moses  42

43. Y. Moses  43

44. Y. Moses  44 Detected Correspondence

45. Y. Moses  45 Error correction + multi-neighbord Multi-neighbors Basic scheme (3 images) Error correction no multi-neighbors

46. Y. Moses  46 New Images Synthetic Images

47. Y. Moses  47 Sec a Ground truth Basic scheme Multi-neighbors Error correction

48. Y. Moses  48 Sec b Ground truth Basic scheme Multi-neighbors Error correction

49. Y. Moses  49 Sec c Ground truth Basic scheme Multi-neighbors Error correction

50. Y. Moses  50 Ground truth Basic scheme Multi-neighbors approx. Error correction Sec d Ground truth Basic scheme Multi-neighbors Error correction

51. Y. Moses  51 Combining Photometry and Geometry Yields a dense correspondence and dense shape recovery of the object in a single path

52. Y. Moses  52 Assumptions  Bilaterally Symmetric object  Lambertian surface with constant albedo  Orthographic projection  Neither occlusions nor shadows  Known “epipolar geometry”

53. Y. Moses  53 Geometric Stereo  2 different images  Known camera parameters  Known viewpoints  Known correspondence 3D shape recovery

54. Y. Moses  54 Computing the Depth from Disparity plpl prpr P qlql qrqr Z Z Orthographic Projection

55. Y. Moses  55 Symmetry and Geometric Stereo Non frontal view of a symmetric object Two different images of the same object

56. Y. Moses  56 Symmetry and Geometric Stereo Non frontal view of a symmetric object Two different images of the same object

57. Y. Moses  57 Geometry  Weak perspective projection: Around X Around Z Around Y

58. Y. Moses  58 Geometry  Projection of R y : Around Y  is the only pose parameter Image point Object point

59. Y. Moses  59 object x z image Correspondence Assume YxZ is the symmetry plane.

60. Y. Moses  60 Mono-Geometric Stereo  3D reconstruction: given correspondence and , unknown known z image x object

61. Y. Moses  61 Viewpoint Invariant  Given the correspondence and unknown  Invariant

62. Y. Moses  62 Photometric Stereo  2 images  Lambertian reflectance  Known illuminations  Known correspondence (same viewpoint) 3D shape recovery

63. Y. Moses  63 Symmetry and Photometric Stereo Non-frontal illumination of a symmetric object Two different images of the same object

64. Y. Moses  64 Notation: Photometry  Corresponding object points:  Illumination:

65. Y. Moses  65 Mono-Photometric Stereo  3D reconstruction given correspondence and E (up to a twofold ambiguity): unknown known

66. Y. Moses  66 Invariance to Illumination  Given correspondence and E unknown  Invariant:

67. Y. Moses  67 Mono-Photometric Stereo  3D reconstruction E unknown but correspondence is given  Frontal viewpoint with non-frontal illumination.  Use image first derivatives.

68. Y. Moses  68 Mono-Photometric Stereo Using image derivatives  3 global unknowns: E  For each pair:  5 unknowns z x z y z xx z xy z yy  6 equations  3 pairs are sufficient

69. Y. Moses  69 Mono-Photometric Stereo Unknown Illumination

70. Y. Moses  70 Correspondence  No correspondence => no stereo.  Hard to define correspondence in images of smooth surfaces.  Almost any correspondence is legal when:  Only geometric constraints are considered.  Only photometric constraints are considered.

71. Y. Moses  71 Combining Photometry and Geometry  Yields a dense correspondence (dense shape recovery of the object).  Enables recovering of the global parameters.

72. Y. Moses  72 Self-Correspondence  A self-correspondence function:

73. Y. Moses  73 Dense Correspondence using Propagation Assume correspondence between a pair of points, p 0 l and p 0 r.

74. Y. Moses  74 Dense Correspondence using Propagation

75. Y. Moses  75 x z image object

76. Y. Moses  76 First derivatives of the Correspondence  Assume known   Assume known E

77. Y. Moses  77 Computing and  Object coordinates: Given computing and is trivial  Moving from object to image coordinates depends on the viewing parameter 

78. Y. Moses  78  Derivatives with respect to the object coordinates:  Derivatives with respect to the image coordinates:

79. Y. Moses  79 x z image object E

80. Y. Moses  80  Given a corresponding pair and E  n=(z x,z y,-1) T  Given  and n  c x and c y  Given c x and c y  a new corresponding pair General Idea    

81. Y. Moses  81 Results on Real Images: Given global parameters

82. Y. Moses  82 Finding Global Parameters  Assume E and  are unknown.  Assume a pair of corresponding points is given.  Two possibilities:  Search for E and  directly.  Compute E and  from the image second derivatives.

83. Y. Moses  83  All roads lead to Rome …  Find and verify correct correspondence  Recover global parameters, E and  Integration Constraint: Circular Tour

84. Y. Moses  84 Finding Global Parameters Consider image second derivatives  Due to foreshortening effect: and  We can relate image and object derivatives by

85. Y. Moses  85 For each corresponding pair: and Plus 4 linear equations in 3 unknown. Where Testing E and  : Image second derivatives

86. Y. Moses  86 Counting  5 unknowns for each pair: z x z y,z xx z xy z yy  4 global unknowns: E,   For each pair: 6 equations.  For n pairs: 5n+4 unknowns 6n equations. 4 pairs are sufficient

87. Y. Moses  87 Results on Simulated Data Ground Truth Recovered Shape

88. Y. Moses  88 Recovering the Global Parameters

89. Y. Moses  89 Degenerate Case  Close to frontal view: problems with geometric-stereo. reconstruction problem  Close to frontal illumination: problems with photometric-stereo. correspondence problem

90. Y. Moses  90 Future work  Perspective photometric stereo  Use as a first approximation to global optimization methods  Test on other reflection models  Recovering of the global parameters:  Light  Cameras  Detect the first pair of correspondence

91. Y. Moses  91 Future Work  Extend to general 3 images under 3 viewpoints and 3 illuminations.  Extend to non-lambertian surfaces.

92. Y. Moses  92 Thanks

93. Y. Moses  93 x z image object

94. Y. Moses  94 Integration Constraint

95. Y. Moses  95 Integration Constraint

96. Y. Moses  96 Searching for E  Illumination must satisfy:   E is further constrained by the image second derivatives.

97. Y. Moses  97 Image second derivatives: Where 4 linear equations in 3 unknown

98. Y. Moses  98 For each corresponding pair and E: 4 linear equations in 3 unknown. Where Image second derivatives

99. Y. Moses  99 Counting  5 unknowns for each pair: z x,z y,z xx,z xy,z yy  3 global unknowns: E  For each pair: 6 equations.  For n pairs: 5n+3 unknowns 6n equations. 3 pairs are sufficient

100. Y. Moses  100 Correspondence

101. Y. Moses  101 Variations  Known/unknown distant light source  Known/unknown viewpoint  Symmetric/non-symmetric image  Frontal/non-frontal viewpoint  Frontal/non-frontal illumination

102. Y. Moses  102 Correspondence  Epipolar geometry is the only geometric constraint on the correspondence.  Weak photometric constraint on the correspondence.

103. Y. Moses  103 Lambertian Surface 5 Basic radiometric I =I = E * * P E E  n2n2  n1n1 

104. Y. Moses  104 E Photometric Stereo  First proposed by Woodham, 1980.  Assume that we have two images..