1. Active Lighting for Appearance Decomposition Todd Zickler DEAS, Harvard University

2. Appearance Decomposition I = f ( shape, reflectance ) Appearance f -1 ( I ) = ? illumination,

3. Appearance Decomposition Research Overview APPEARANCE CAPTURE COLOR IMAGE FILTERING 3D RECONSTRUCTION PHOTOMETRIC INVARIANTS

4. Appearance Decomposition Getting 3D Shape: Image-based Reconstruction I = f ( shape, reflectance, illumination ) f -1 ( I ) = ?

5. Appearance Decomposition Reflectance: BRDF Bi-directional Reflectance Distribution Function

6. Appearance Decomposition Conventional 3D Reconstruction: Restrictive Assumptions LAMBERTIAN: IDEALLY DIFFUSE

7. Appearance Decomposition Example: Conventional Stereo ASSUMPTION: I l = I r IlIl IrIr

8. Appearance Decomposition Example: Conventional Stereo IlIl IrIr ASSUMPTION: I l = I r

9. Appearance Decomposition Conventional 3D Reconstruction: Restrictive Assumptions Shape from shading [Tsai and Shaw, 1994] Variational Stereo [Faugeras and Keriven, 1998] Space Carving [Kutulakos and Seitz, 1998] Multiple-window stereo [Fusiello et al., 1997]

10. Appearance Decomposition Reflectance: BRDF

11. Appearance Decomposition Reflectance: BRDF

12. Appearance Decomposition Helmholtz Reciprocity [Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]

13. Appearance Decomposition Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

14. Appearance Decomposition Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

15. Appearance Decomposition Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO

16. Appearance Decomposition Reciprocal Images  Specularities “fixed” to surface IlIl IrIr  Relation between I l and I r independent of BRDF

17. Appearance Decomposition Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n

18. Appearance Decomposition Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n  Arbitrary reflectance  Surface normal

19. Appearance Decomposition Reciprocal Acquisition CAMERA LIGHT SOURCE

20. Appearance Decomposition Recovered Normals [Zickler et al. 2002]

21. Appearance Decomposition Recovered Surface [Zickler et al., ECCV 2002]

22. Appearance Decomposition In Practice 1.Arbitrary Reflectance 2.Off-the-shelf components 3.Direct surface normals 4.Images aligned with recovered shape 5.Self-calibrating (coming…)

23. Appearance Decomposition Ongoing Work: Auto-calibration [Zickler et al., CVPR 2003, CVPR 2006,…]

24. Appearance Decomposition Research Overview APPEARANCE CAPTURE COLOR IMAGE FILTERING 3D RECONSTRUCTION PHOTOMETRIC INVARIANTS

25. Appearance Decomposition Reflectance Decomposition DIFFUSE =+ SPECULAR [Phong 1975; Shafer, 1985]

26. Appearance Decomposition Reflectance Decomposition [Shafer, 1985]

27. Appearance Decomposition Reflectance Decomposition: Simplifies the Vision Problem =+ LAMBERTIAN: IDEALLY DIFFUSE =+

28. Appearance Decomposition Reflectance Decomposition: A Difficult Inverse Problem DIFFUSE =+ SPECULAR [Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991,…] =+

29. Appearance Decomposition Known Illuminant: Still Ill-posed G S I RGB B R D?

30. Appearance Decomposition Known Illuminant: Still Ill-posed G S I RGB B R D?

31. Appearance Decomposition Observation: Explicit Decomposition not Required G r2r2 r1r1 S I RGB B R J 1.INVARIANT TO SPECULAR REFLECTIONS 2.BEHAVES ‘LAMBERTIAN’

32. Appearance Decomposition Observation: Explicit Decomposition not Required G r2r2 r1r1 S I RGB B R J || J || [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]

33. Appearance Decomposition Generalization: Mixed Illumination G r2r2 r1r1 S I RGB B R J r1r1 S1S1 B G R S2S2 J SINGLE ILLUMINANTMIXED ILLUMINATION [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]

34. Appearance Decomposition Generalization: Mixed Illumination

35. Appearance Decomposition Example: Binocular Stereo [Algorithm: Boykov, Veksler and Zabih, CVPR 1998] Conventional Grayscale (R+G+B)/3 Specular Invariant, ||J|| (blue illuminant) Specular Invariant, ||J|| (blue & yellow illuminants) One image from input stereo pair Recovered depth

36. Appearance Decomposition Example: Optical Flow [Algorithm: Black and Anandan, 1993] Conventional Grayscale (R-+G+B)/3 Specular Invariant, ||J|| (blue illuminant) Specular Invariant, ||J|| (blue & yellow illuminants) Ground truth flow

37. Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] J behaves ‘Lambertian’  Linear function of surface normal

38. Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] J behaves ‘Lambertian’  Linear function of surface normal

39. Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] J behaves ‘Lambertian’  Linear function of surface normal

40. Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]

41. Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]

42. Appearance Decomposition Generalized Hue G r2r2 r1r1 S I RGB B R J ψ

43. Appearance Decomposition Example: Material-based Segmentation [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Input image Conventional GrayscaleSpecular Invariant ||J|| Conventional Hue Generalized Hue 

44. Appearance Decomposition Active lighting can provide: 1. Precise shape (surface normals) for a broad class of (non-Lambertian) surfaces 2. Specular and/or shading invariance (e.g., optical flow, tracking, segmentation) 3. Minimal hardware requirements Active Lighting for Image-guided Surgery? Endoscopic imagery: 1. Illuminant(s) is/are controlled and known 2. Non-Lambertian surfaces 3. Lack of texture

45. Appearance Decomposition Acknowledgements Satya Mallick, UCSD Peter Belhumeur, Columbia University David Kriegman, UCSD Sebastian Enrique, Columbia University Ravi Ramamoorthi, Columbia University zickler@eecs.harvard.edu http://www.eecs.harvard.edu/~zickler