1. Color and Brightness Constancy Jim Rehg CS 4495/7495 Computer Vision Lecture 25 & 26 Wed Oct 18, 2002

2. J. M. Rehg © 2002 2 Outline  Human color inference  Land’s Retinex  Dichromatic reflectance model  Finite dimensional linear models  Color constancy algorithm

3. J. M. Rehg © 2002 3 Human Color Constancy  Distinguish between Color constancy, which refers to hue and saturation Color constancy, which refers to hue and saturation Lightness constancy, which refers to gray-level. Lightness constancy, which refers to gray-level.  Humans can perceive Color a surface would have under white light (surface color) Color a surface would have under white light (surface color) Color of the reflected light (limited ability to separate surface color from measured color) Color of the reflected light (limited ability to separate surface color from measured color) Color of illuminant (even more limited) Color of illuminant (even more limited)

4. J. M. Rehg © 2002 4 Spatial Arrangement and Color Perception

5. J. M. Rehg © 2002 5 Spatial Arrangement and Color Perception

6. J. M. Rehg © 2002 6 Spatial Arrangement and Color Perception

7. J. M. Rehg © 2002 7 Land’s Mondrian Experiments  The (by-now) familiar phenomena: Squares of color with the same color radiance yield very different color perceptions Photometer: 1.0, 0.3, 0.3 Audience: “Red”Audience: “Blue” White light Colored light

8. J. M. Rehg © 2002 8 Basic Model for Lightness Constancy  Modeling assumptions for camera Planar frontal scene Planar frontal scene Lambertian reflectance Lambertian reflectance Linear camera response Linear camera response  Camera model:  Modeling assumptions for scene Albedo is piecewise constant Albedo is piecewise constant – Exception: ripening fruit Illumination is slowly-varying Illumination is slowly-varying – Exception: shadow boundaries

9. J. M. Rehg © 2002 9 Algorithm Components  The goal is to determine what the surfaces in the image would look like under white light.  A process that compares the brightness of patchs across their common boundaries and computes relative brightness.  A process that establishes an absolute reference for lightness (e.g. brightest point is “white”)

10. J. M. Rehg © 2002 10 1-D Lightness “Retinex” Threshold gradient image to find surface (patch) boundaries

11. J. M. Rehg © 2002 11 1-D Lightness “Retinex” Integration to recover surface lightness (unknown constant)

12. J. M. Rehg © 2002 12 Extension to 2-D  Spatial issues Integration becomes much harder Integration becomes much harder – Integrate along many sample paths (random walk) – Loopy propagation  Recover of absolute lightness/color reference Brightest patch is white Brightest patch is white Average reflectance across scene is known Average reflectance across scene is known Gamut is known Gamut is known Specularities can be detected Specularities can be detected Known reference (color chart, skin color, etc.) Known reference (color chart, skin color, etc.)

13. J. M. Rehg © 2002 13 Color Retinex Images courtesy John McCann

14. J. M. Rehg © 2002 14 Finding Specularities  Dielectric materials Specularly reflected light has the color of the source Specularly reflected light has the color of the source  Reflected light has two components, we see their sum Diffuse (body reflection) Diffuse (body reflection) Specular (highlight) Specular (highlight)  Specularities produce a “Skewed-T” in the color histogram of the object.

15. J. M. Rehg © 2002 15 Skewed-T in Histogram A Physical Approach to Color Image Understanding – Klinker, Shafer, and Kanade. IJCV 1990

16. J. M. Rehg © 2002 16 Skewed-T in Histogram

17. J. M. Rehg © 2002 17 Recent Application to Stereo Motion of camera causes highlight location to change. This cue can be combined with histogram analysis. Synthetic scene:

18. J. M. Rehg © 2002 18 Recent Application to Stereo “Real” scene:

19. J. M. Rehg © 2002 19 Finite Dimensional Linear Models

20. J. M. Rehg © 2002 20 Obtaining the illuminant from specularities  Assume that a specularity has been identified, and material is dielectric.  Then in the specularity, we have  Assuming we know the sensitivities and the illuminant basis functions we know the sensitivities and the illuminant basis functions there are no more illuminant basis functions than receptors there are no more illuminant basis functions than receptors  This linear system yields the illuminant coefficients.

21. J. M. Rehg © 2002 21 Obtaining the illuminant from average color assumptions  Assume the spatial average reflectance is known  We can measure the spatial average of the receptor response to get  Assuming g_ijk are known g_ijk are known average reflectance is known average reflectance is known there are not more receptor types than illuminant basis functions there are not more receptor types than illuminant basis functions  We can recover the illuminant coefficients from this linear system

22. J. M. Rehg © 2002 22 Normalizing the Gamut  The gamut (collection of all pixel values in image) contains information about the light source It is usually impossible to obtain extreme color readings (255,0,0) under white light It is usually impossible to obtain extreme color readings (255,0,0) under white light  The convex hull of the gamut constrains illuminant  Gamut mapping algorithm (Forsyth ’90) Obtain convex hull W of pixels under white light Obtain convex hull W of pixels under white light Obtain convex hull G of input image Obtain convex hull G of input image The mapping M(G) must have property The mapping M(G) must have property