1. Color, lightness & brightness Lavanya Sharan February 7, 2011

2. Same or different cubes?

3. Demonstration by Purves et al. (2002) Image source: https://engineering.purdue.edu/~bouman/ece637/notes/ColorConstancy/color/ You don’t see raw pixel values

4. Demonstration by Purves et al. (2002) Image source: https://engineering.purdue.edu/~bouman/ece637/notes/ColorConstancy/color/ Color contrast: perceived color depends on context

5. Demonstration by Purves et al. (2002) Image source: https://engineering.purdue.edu/~bouman/ece637/notes/ColorConstancy/color/ Color constancy: perceived color context-independent

6. Which color do we see? Don’t trust pixel values It is all relative Trust pixel values It is all absolute

7. Which color do we see? Don’t trust pixel values It is all relative Trust pixel values It is all absolute Same pixel values can look as if two different colors Different pixels values can look as if same color

8. Which color do we see? Don’t trust pixel values It is all relative Trust pixel values It is all absolute Same pixel values can look as if two different colors Different pixels values can look as if same color Phenomenon: Color contrast Phenomenon: Color constancy

9. Image source: Brainard et al. (2006) R: 197 G: 194 B: 160 R: 137 G: 157 B: 148 What happens in the real world?

10. AB A’B’ Observation: A’ doesn’t look exactly like B’, but this difference seems smaller than A vs. B. A bit of both (contrast and constancy) is going on.

11. Questions ‣ Contrast and constancy are color phenomena. What about mechanisms that can explain the phenomena? ‣ How can we study and measure color perception mechanisms? ‣ How much do we know? What are the open questions?

12. (Postulated) mechanisms of color perception Contrast coding ‣ L, M, S cone responses (remember Lec 2?) are inputs ‣ Local contrast is encoded for each cone response ‣ Explains invariance to illumination intensity changes ‣ Does not explain effects of reflectance changes surrounding test surface rLrL uLuL c L = (r L -u L )/u L Slide content: Brainard (2010)

13. (Postulated) mechanisms of color perception Beyond contrast coding ‣ Mechanistic approaches ‣ Transformations of the L, M, S response through various stages. ‣ Connection to physiology, in the limit explain “each neuron”. ‣ Try to explain a wide range of behavioral data. Slide content: Brainard (2010), Stockman & Brainard (2010)

14. (Postulated) mechanisms of color perception Beyond contrast coding ‣ Computational approaches ‣ More like computer vision, estimate illumination and reflectance (make assumptions about shape) ‣ Ignore details of human vision ‣ Compare output of computational model to behavioral data and ground truth (physically correct parameters). Slide content: Brainard (2010) Image source: Brainard et al. (2006)

15. Methods for studying color perception Test constancy by varying: Illumination characteristics Surface shape Surface reflectance properties Background Varying surface gloss (Xiao & Brainard 2008)

16. Methods for studying color perception Varying chromaticity of light sources (Boyaci et al. 2004) Test constancy by varying: Illumination characteristics Surface shape Surface reflectance properties Background

17. Methods for studying color perception Varying illumination cues such as cast shadows, shading and specular highlights (Boyaci et al. 2006) Test constancy by varying: Illumination characteristics Surface shape Surface reflectance properties Background

18. Color perception: What is known Akiyoshi Kitaoka (2005) same Lots of color effects for flat, uniform color stimuli Rollers, Akiyoshi Kitaoka (2004)

19. Color perception: What is known When reflectance and illumination changes are confounded, biggest failures of constancy. Akiyoshi Kitaoka (2005) Edges due to illumination change or reflectance change?

20. Color perception: What is known These confounds are less common in real world images, so better constancy (not as well studied as simple stimuli). This is an illumination change or camera response change (Mather 2008) This is a surface reflectance change (http://familyvolley.blogspot.com)

21. Color perception: What is not known What happens in real world images? Complex 3D shape, reflectance properties such as gloss, transparency etc. What is the effect of scene layout? Depth, spatial layout, inter-reflections etc. Which models work? When and why? Inverse optics (solve illumination then color), image statistics (simple, dirty measurements), intrinsic images (decompose in illumination, reflectance and shading) etc... The role of long-term memory? How does prior experience affect color constancy? What about semantic knowledge about objects? Slide courtesy: Bei Xiao

22. Color isn’t all that we see We can function in a grayscale world. A whole other field within visual perception that focuses on perceived luminance and perceived reflectance. Brightness = perceived luminance Lightness = perceived diffuse reflectance

23. Lightness vs. brightness Not the same. Adelson (1993) Lightness: Perception of a physical property in the world. Brightness: Perception of what falls on our sensors.

24. Lightness vs. brightness Not the same. Adelson (1993) Lightness: Perception of a physical property in the world. Brightness: Perception of what falls on our sensors. a & b same reflectance, different orientation => different luminance values a appears brighter than b b & c different reflectance, same orientation => different luminance values c appears brighter and lighter than b

25. Lightness vs. brightness Not the same. Adelson (1993) The distinction can be confusing when your world consists of flat, uniform grayscale blocks! a & b same reflectance, different orientation => different luminance values a appears brighter than b b & c different reflectance, same orientation => different luminance values c appears brighter and lighter than b

26. Context and scene interpretation matters 3D scene, A & B look like tiles of differing reflectance 3D scene interpretation broken to reveal pixel values. A and B look more similar. Cognitively, you know A = B. Like color, simple contrast models are not sufficient.

27. Gilchrist et al. (1999)

29. Gilchrist et al.’s anchoring theory Rule 1: Highest luminance is seen as white

30. Gilchrist et al.’s anchoring theory Rule 1I : Largest area is seen as white

31. Gilchrist et al.’s anchoring theory How to apply these rules: concept of frameworks Global framework Local framework Room in which subject is seated. Test squares are brightly lit and rest of room is dark.

32. Gilchrist et al.’s anchoring theory Effects of configuration (arrangement of squares), articulation (number of squares), and field size (size of squares) on perceived reflectance. Room in which subject is seated. Test squares are brightly lit and rest of room is dark.

33. What about real world images? Do effects measured with simple stimuli generalize? Classic Gelb Effect (1929): Two discs look the same.

34. What about real world images? Do effects measured with simple stimuli generalize? Not always. Anti-Gelb Demo: The two surfaces look very different. (Sharan et al. 2008)

35. What about real world images? Do effects measured with simple stimuli generalize? Not always. For isolated surfaces like these, simple image statistics are correlated with human performance (Sharan et al. 2008, Motoyoshi et al. 2007)

36. Lightness perception: What is known Lots of effects, mainly on simple, ‘Mondrian world’ stimuli. We can reliably estimate reflectance, we do better with more information in the stimuli. Grouping effects are important (i.e., context and scene interpretation).

37. Lightness perception: What is not known What happens in real world images? Complex 3D shape, reflectance properties such as gloss, transparency etc. Which models work? When and why? Inverse optics (solve illumination then reflectance), image statistics (simple, dirty measurements), intrinsic images (decompose in illumination, reflectance and shading), for grouping-based models how to compute frameworks? Connection to texture? For complex surfaces, shape and reflectance combine to form “3-D textures” (Pont & Koenderink 2005). How do we study those?

38. Summary ✓ Long history of studying color, lightness and brightness. See assigned readings for reviews (ask me for more). ✓ Most work has looked at simple, flat 2D stimuli with uniform, matte colors. ✓ This work has led to lots of cool illusions :-) ✓ However, real world is more complex and we are only now beginning to study effects in real world situations. ✓ Lots of parallel computational work in CVG (e.g, BRDF estimation, color correction, object recognition features etc.) that can serve as models for human perception.