Y. Moses 11 Combining Photometric and Geometric Constraints Yael Moses IDC, Herzliya Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses 22 Recover the 3D shape of a general smooth surface from a set of calibrated images Problem 1:

Y. Moses 33 Problem 2: Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses 44 Shape Recovery  Geometry: Stereo  Photometry:  Shape from shading  Photometric stereo Main problems: Calibrations and Correspondence

Y. Moses 55 3D Shape Recovery Photometry:  Shape from shading  Photometric stereo Geometry:  Stereo  Structure from motion

Y. Moses 66 Geometric Stereo  2 different images  Known camera parameters  Known correspondence + +

Y. Moses 77 Photometric Stereo  3D shape recovery: surface normals from two or more images taken from the same viewpoint

Y. Moses 88 Photometric Stereo Solution: Three images Matrix notation

Y. Moses 99 Photometric Stereo  3D shape recovery (surface normals) Two or more images taken from the same viewpoint Main Limitation: Correspondence is obtained by a fixed viewpoint

Y. Moses  10 Overview  Combining photometric and geometric stereo:  Symmetric surface, single image  Non symmetric: 3 images  Mono-Geometric stereo  Mono-Photometric stereo  Experimental results.

Y. Moses  11 The input  Smooth featureless surface  Taken under different viewpoints  Illuminated by different light sources The Problem: Recover the 3D shape from a set of calibrated images

Y. Moses  12 Assumptions  Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …) * *  n n  Three or more images  Perspective projection

Y. Moses  13 Our method Combines photometric and geometric stereo We make use of:  Given Correspondence:  Can compute a normal  Can compute the 3D point

Y. Moses  14 Basic Method Given Correspondence

Y. Moses  15 First Order Surface Approximation

Y. Moses  16 First Order Surface Approximation

Y. Moses  17 First Order Surface Approximation P(  ) = (1 -  )O 1 +  P , N (P(  ) - P) = 0

Y. Moses  18 First Order Surface Approximation

Y. Moses  19 New Correspondence

Y. Moses  20 New Surface Approximation

Y. Moses  21 Dense Correspondence

Y. Moses  22 Basic Propagation

Y. Moses  23 Basic Propagation

Y. Moses  24 Basic method: First Order  Given correspondence p i and L  P and n  Given P and n  T  Given P, T and M i  a new correspondence q i

Y. Moses  25 Extensions  Using more than three images  Propagation:  Using multi-neighbours  Smart propagation  Second error approximation  Error correction:  Based on local continuity  Other assumptions on the surface

Y. Moses  26 Multi-neighbors Propagation

Y. Moses  27 Smart Propagation

Y. Moses  28 Second Order: a Sphere P()P() N+N  N NN P (P-P(  ))(N+N  )=0

Y. Moses  29 Second Order Approximation

Y. Moses  30 Second Order Approximation

Y. Moses  31 Using more than three images  Reduce noise of the photometric stereo  Avoid shadowed pixels  Detect “bad pixels”  Noise  Shadows  Violation of assumptions on the surface

Y. Moses  32 Smart Propagation

Y. Moses  33 Error correction The compatibility of the local 3D shape can be used to correct errors of:  Correspondence  Camera parameters  Illumination parameters

Y. Moses  34 Score  Continuity:  Shape  Normals  Albedo  The consistency of 3D points locations and the computed normals:  General case: full triangulation  Local constraints

Y. Moses  35 Extensions  Using more than three images  Propagation:  Using multi-neighbours  Smart propagation  Second error approximation  Error correction:  Based on local continuity  Other assumptions on the surface

Y. Moses  36 Real Images  Camera calibration  Light calibration  Direction  Intensity  Ambient

Y. Moses  37 Error correction + multi-neighbor 5 Images

Y. Moses  38 5pp 3pp 3nn 5nn 5pn

Y. Moses  39

Y. Moses  40

Y. Moses  41

Y. Moses  42

Y. Moses  43

Y. Moses  44 Detected Correspondence

Y. Moses  45 Error correction + multi-neighbord Multi-neighbors Basic scheme (3 images) Error correction no multi-neighbors

Y. Moses  46 New Images Synthetic Images

Y. Moses  47 Sec a Ground truth Basic scheme Multi-neighbors Error correction

Y. Moses  48 Sec b Ground truth Basic scheme Multi-neighbors Error correction

Y. Moses  49 Sec c Ground truth Basic scheme Multi-neighbors Error correction

Y. Moses  50 Ground truth Basic scheme Multi-neighbors approx. Error correction Sec d Ground truth Basic scheme Multi-neighbors Error correction

Y. Moses  51 Combining Photometry and Geometry Yields a dense correspondence and dense shape recovery of the object in a single path

Y. Moses  52 Assumptions  Bilaterally Symmetric object  Lambertian surface with constant albedo  Orthographic projection  Neither occlusions nor shadows  Known “epipolar geometry”

Y. Moses  53 Geometric Stereo  2 different images  Known camera parameters  Known viewpoints  Known correspondence 3D shape recovery

Y. Moses  54 Computing the Depth from Disparity plpl prpr P qlql qrqr Z Z Orthographic Projection

Y. Moses  55 Symmetry and Geometric Stereo Non frontal view of a symmetric object Two different images of the same object

Y. Moses  56 Symmetry and Geometric Stereo Non frontal view of a symmetric object Two different images of the same object

Y. Moses  57 Geometry  Weak perspective projection: Around X Around Z Around Y

Y. Moses  58 Geometry  Projection of R y : Around Y  is the only pose parameter Image point Object point

Y. Moses  59 object x z image Correspondence Assume YxZ is the symmetry plane.

Y. Moses  60 Mono-Geometric Stereo  3D reconstruction: given correspondence and , unknown known z image x object

Y. Moses  61 Viewpoint Invariant  Given the correspondence and unknown  Invariant

Y. Moses  62 Photometric Stereo  2 images  Lambertian reflectance  Known illuminations  Known correspondence (same viewpoint) 3D shape recovery

Y. Moses  63 Symmetry and Photometric Stereo Non-frontal illumination of a symmetric object Two different images of the same object

Y. Moses  64 Notation: Photometry  Corresponding object points:  Illumination:

Y. Moses  65 Mono-Photometric Stereo  3D reconstruction given correspondence and E (up to a twofold ambiguity): unknown known

Y. Moses  66 Invariance to Illumination  Given correspondence and E unknown  Invariant:

Y. Moses  67 Mono-Photometric Stereo  3D reconstruction E unknown but correspondence is given  Frontal viewpoint with non-frontal illumination.  Use image first derivatives.

Y. Moses  68 Mono-Photometric Stereo Using image derivatives  3 global unknowns: E  For each pair:  5 unknowns z x z y z xx z xy z yy  6 equations  3 pairs are sufficient

Y. Moses  69 Mono-Photometric Stereo Unknown Illumination

Y. Moses  70 Correspondence  No correspondence => no stereo.  Hard to define correspondence in images of smooth surfaces.  Almost any correspondence is legal when:  Only geometric constraints are considered.  Only photometric constraints are considered.

Y. Moses  71 Combining Photometry and Geometry  Yields a dense correspondence (dense shape recovery of the object).  Enables recovering of the global parameters.

Y. Moses  72 Self-Correspondence  A self-correspondence function:

Y. Moses  73 Dense Correspondence using Propagation Assume correspondence between a pair of points, p 0 l and p 0 r.

Y. Moses  74 Dense Correspondence using Propagation

Y. Moses  75 x z image object

Y. Moses  76 First derivatives of the Correspondence  Assume known   Assume known E

Y. Moses  77 Computing and  Object coordinates: Given computing and is trivial  Moving from object to image coordinates depends on the viewing parameter 

Y. Moses  78  Derivatives with respect to the object coordinates:  Derivatives with respect to the image coordinates:

Y. Moses  79 x z image object E

Y. Moses  80  Given a corresponding pair and E  n=(z x,z y,-1) T  Given  and n  c x and c y  Given c x and c y  a new corresponding pair General Idea    

Y. Moses  81 Results on Real Images: Given global parameters

Y. Moses  82 Finding Global Parameters  Assume E and  are unknown.  Assume a pair of corresponding points is given.  Two possibilities:  Search for E and  directly.  Compute E and  from the image second derivatives.

Y. Moses  83  All roads lead to Rome …  Find and verify correct correspondence  Recover global parameters, E and  Integration Constraint: Circular Tour

Y. Moses  84 Finding Global Parameters Consider image second derivatives  Due to foreshortening effect: and  We can relate image and object derivatives by

Y. Moses  85 For each corresponding pair: and Plus 4 linear equations in 3 unknown. Where Testing E and  : Image second derivatives

Y. Moses  86 Counting  5 unknowns for each pair: z x z y,z xx z xy z yy  4 global unknowns: E,   For each pair: 6 equations.  For n pairs: 5n+4 unknowns 6n equations. 4 pairs are sufficient

Y. Moses  87 Results on Simulated Data Ground Truth Recovered Shape

Y. Moses  88 Recovering the Global Parameters

Y. Moses  89 Degenerate Case  Close to frontal view: problems with geometric-stereo. reconstruction problem  Close to frontal illumination: problems with photometric-stereo. correspondence problem

Y. Moses  90 Future work  Perspective photometric stereo  Use as a first approximation to global optimization methods  Test on other reflection models  Recovering of the global parameters:  Light  Cameras  Detect the first pair of correspondence

Y. Moses  91 Future Work  Extend to general 3 images under 3 viewpoints and 3 illuminations.  Extend to non-lambertian surfaces.

Y. Moses  92 Thanks

Y. Moses  93 x z image object

Y. Moses  94 Integration Constraint

Y. Moses  95 Integration Constraint

Y. Moses  96 Searching for E  Illumination must satisfy:   E is further constrained by the image second derivatives.

Y. Moses  97 Image second derivatives: Where 4 linear equations in 3 unknown

Y. Moses  98 For each corresponding pair and E: 4 linear equations in 3 unknown. Where Image second derivatives

Y. Moses  99 Counting  5 unknowns for each pair: z x,z y,z xx,z xy,z yy  3 global unknowns: E  For each pair: 6 equations.  For n pairs: 5n+3 unknowns 6n equations. 3 pairs are sufficient

Y. Moses  100 Correspondence

Y. Moses  101 Variations  Known/unknown distant light source  Known/unknown viewpoint  Symmetric/non-symmetric image  Frontal/non-frontal viewpoint  Frontal/non-frontal illumination

Y. Moses  102 Correspondence  Epipolar geometry is the only geometric constraint on the correspondence.  Weak photometric constraint on the correspondence.

Y. Moses  103 Lambertian Surface 5 Basic radiometric I =I = E * * P E E  n2n2  n1n1 

Y. Moses  104 E Photometric Stereo  First proposed by Woodham,  Assume that we have two images..