1. Image-Based Modeling of Complex Surfaces Todd Zickler DEAS, Harvard University

2. Image-Based Modeling of Complex Surfaces Appearance

3. Image-Based Modeling of Complex Surfaces Modeling Appearance: Face Recognition [Harvard Face Database]

4. Image-Based Modeling of Complex Surfaces Modeling Appearance: Tracking Spacetime Faces [Zhang et al., 2004] Detection/tracking of facial features [Colmenarez et al., 1999]

5. Image-Based Modeling of Complex Surfaces Modeling Appearance: Applications  Object recognition  Face recognition  Visual tracking  Human-computer interfaces  Compression  Navigation  Surveillance  Metrology  Inspection  Vision for Graphics

6. Image-Based Modeling of Complex Surfaces Appearance

7. Image-Based Modeling of Complex Surfaces Non-parametric Appearance: Uniform Sampling? 5 º sampling 1,000,000 images >1x10 6 Mb 1 º sampling 625,000,000 images >1x10 9 Mb

8. Image-Based Modeling of Complex Surfaces Appearance = Shape + Reflectance SHAPEREFLECTANCE

9. Image-Based Modeling of Complex Surfaces Reflectance

10. Image-Based Modeling of Complex Surfaces Outline I = f ( shape, reflectance, illumination ) 1. SHAPE f -1 ( I ) = ?

11. Image-Based Modeling of Complex Surfaces Outline 1. SHAPE I = f ( shape, reflectance, illumination )

12. Image-Based Modeling of Complex Surfaces Outline 1. SHAPE + 2. REFLECTANCE

13. Image-Based Modeling of Complex Surfaces Shape: Restrictive Reflectance Assumptions Shape from shading [Tsai and Shaw, 1994] Variational Stereo [Faugeras and Keriven, 1998] Space Carving [Kutulakos and Seitz, 1998] Multiple-window stereo [Fusiello et al., 1997]

14. Image-Based Modeling of Complex Surfaces Simple (and Common!) Reflectance Model LAMBERTIAN: IDEALLY DIFFUSE

15. Image-Based Modeling of Complex Surfaces Example: Conventional Stereo

16. Image-Based Modeling of Complex Surfaces Example: Conventional Stereo

17. Image-Based Modeling of Complex Surfaces Reflectance: BRDF

18. Image-Based Modeling of Complex Surfaces Reflectance: BRDF

19. Image-Based Modeling of Complex Surfaces Helmholtz Reciprocity [Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]

20. Image-Based Modeling of Complex Surfaces Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

21. Image-Based Modeling of Complex Surfaces Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

22. Image-Based Modeling of Complex Surfaces Reciprocal Images  Specularities “fixed” to surface IlIl IrIr  Relation between I l and I r independent of BRDF

23. Image-Based Modeling of Complex Surfaces Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n

24. Image-Based Modeling of Complex Surfaces Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n  Arbitrary reflectance  Surface normal

25. Image-Based Modeling of Complex Surfaces Reciprocal Acquisition CAMERA LIGHT SOURCE

26. Image-Based Modeling of Complex Surfaces Recovered Normals [ECCV 2002]

27. Image-Based Modeling of Complex Surfaces Recovered Surface [ECCV 2002]

28. Image-Based Modeling of Complex Surfaces Helmholtz Stereopsis: Recent Work Uncalibrated Helmholtz Stereopsis Binocular Helmholtz Stereopsis [CVPR 2003][ICCV 2003]

29. Image-Based Modeling of Complex Surfaces Outline 1. SHAPE + 2. REFLECTANCE

30. Image-Based Modeling of Complex Surfaces Spatially-varying Reflectance 5 º sampling: 1,000,000 images; >1x10 6 Mb 1 º sampling: 625,000,000 images; >1x10 9 Mb x 

31. Image-Based Modeling of Complex Surfaces Point-wise Reflectance Measurement (& Light Fields) Stanford Computer Graphics LaboratoryUSC Institute for Creative Technologies Graphics Lab Mitsubishi Electric Research LaboratoriesUniversity of Illinois Beckman Institute

32. Image-Based Modeling of Complex Surfaces Reflectance Sharing = Scattered Data Interpolation )()θ,(qfxf     q  x θ  

33. Image-Based Modeling of Complex Surfaces Application: Human Face

34. Image-Based Modeling of Complex Surfaces Application: Human Face DIFFUSESPECULAR          N i iikkk qqqplxaqf 1 λ)()()( ~ 

35. Image-Based Modeling of Complex Surfaces PREDICTED Application: Human Face ACTUALPREDICTED

36. Image-Based Modeling of Complex Surfaces Predicting Appearance in Real Time

37. Image-Based Modeling of Complex Surfaces Conclusion SHAPEREFLECTANCE

38. Image-Based Modeling of Complex Surfaces Future Work 1. Face/object recognition 2. Surface structure at shape/reflectance interface, transparency 3. Appearance over time  weathering, aging 4. Dynamic appearance  Reflectance under deformation

39. zickler@eecs.harvard.edu http://www.eecs.harvard.edu/~zickler