1. Computer Vision - Color Hanyang University Jong-Il Park

2. Department of Computer Science and Engineering, Hanyang University Topics to be covered Light and Color Color Representation Color Discrimination Application

3. Department of Computer Science and Engineering, Hanyang University The visible light spectrum We “see” electromagnetic radiation in a range of wavelengths

4. Department of Computer Science and Engineering, Hanyang University Relative sizes

5. Department of Computer Science and Engineering, Hanyang University Light spectrum The appearance of light depends on its power spectrum  How much power (or energy) at each wavelength daylighttungsten bulb Our visual system converts a light spectrum into “color”  This is a rather complex transformation

6. Department of Computer Science and Engineering, Hanyang University The human visual system Color perception  Light hits the retina, which contains photosensitive cells  rods and cones  These cells convert the spectrum into a few discrete values

7. Department of Computer Science and Engineering, Hanyang University Density of rods and cones Rods and cones are non-uniformly distributed on the retina  Rods responsible for intensity, cones responsible for color  Fovea - Small region (1 or 2°) at the center of the visual field containing the highest density of cones (and no rods).  Less visual acuity in the periphery—many rods wired to the same neuron light ConeRod Retina

8. Department of Computer Science and Engineering, Hanyang University 8 Rods: Twilight Vision 130 million rod cells per eye. 1000 times more sensitive to light than cone cells. Most to green light (about 550-555 nm), but with a broad range of response throughout the visible spectrum. Produces relatively blurred images, and in shades of gray. Pure rod vision is also called twilight vision. Relative neural response of rods as a function of light wavelength. 400500600700 Wavelength (nm) 1.00 0.75 0.50 0.25 0.00 Relative response

9. Department of Computer Science and Engineering, Hanyang University 9 Cones: Color Vision 7 million cone cells per eye. Three types of cones* (S, M, L), each "tuned" to different maximum responses at:-  S : 430 nm (blue) (2%)  M: 535 nm (green) (33%)  L : 590 nm (red) (65%) Produces sharp, color images. Pure cone vision is called photopic or color vision. Spectral absorption of light by the three cone types 400500600700 Wavelength (nm) 1.00 0.75 0.50 0.25 0.00 Relative absorbtion S M L *S = Short wavelength cone M = Medium wavelength cone L = Long wavelength cone

10. Department of Computer Science and Engineering, Hanyang University Color perception Three types of cones  Each is sensitive in a different region of the spectrum  Different sensitivities: we are more sensitive to green than red  varies from person to person (and with age)  Colorblindness—deficiency in at least one type of cone L response curve

11. Department of Computer Science and Engineering, Hanyang University Color perception Rods and cones act as filters on the spectrum  To get the output of a filter, multiply its response curve by the spectrum, integrate over all wavelengths  Each cone yields one number  Q: How can we represent an entire spectrum with 3 numbers? S ML Wavelength Power  A: We can’t! Most of the information is lost.  As a result, two different spectra may appear indistinguishable such spectra are known as metamers http://www.cs.brown.edu/exploratories/freeSoftware/repository/edu/brown/cs/explo ratories/applets/spectrum/metamers_guide.htmlhttp://www.cs.brown.edu/exploratories/freeSoftware/repository/edu/brown/cs/explo ratories/applets/spectrum/metamers_guide.html

12. Department of Computer Science and Engineering, Hanyang University Eye Color Sensitivity Although cone response is similar for the L, M, and S cones, the number of the different types of cones vary. L:M:S = 40:20:1 Cone responses typically overlap for any given stimulus, especially for the M-L cones. The human eye is most sensitive to green light. Spectral absorption of light by the three cone types 400500600700 Wavelength (nm) 1.00 0.75 0.50 0.25 0.00 Relative absorbtion S M L S, M, and L cone distribution in the fovea Effective sensitivity of cones (log plot) 400500600700 Wavelength (nm) 1.00 0.1 0.01 0.001 0.0001 Relative sensitivity S M L

13. Department of Computer Science and Engineering, Hanyang University Theory of Trichromatic Vision The principle that the color you see depends on signals from the three types of cones (L, M, S). The principle that visible color can be mapped in terms of the three colors (R, G, B) is called trichromacy. The three numbers used to represent the different intensities of red, green, and blue needed are called tristimulus values. = Tristimulus values r gb

14. Department of Computer Science and Engineering, Hanyang University Seeing Colors The colors we perceive depends on:- Illumination source  Illumination source Object reflectance factor  Object reflectance Observer spectral sensitivity  Observer response Observer response = Tristimulus values (Viewer response) r gb x x  The product of these three factors will produce the sensation of color.

15. Department of Computer Science and Engineering, Hanyang University Additive Colors Start with Black – absence of any colors. The more colors added, the brighter it gets. Color formation by the addition of Red, Green, and Blue, the three primary colors Examples of additive color usage:-  Human eye  Lighting  Color monitors  Color video cameras Additive color wheel

16. Department of Computer Science and Engineering, Hanyang University Subtractive Colors Starts with a white background (usually paper). Use Cyan, Magenta, and/or Yellow dyes to subtract from light reflected by paper, to produce all colors. Examples of Subtractive color use:-  Color printers  Paints Subtractive color wheel

17. Department of Computer Science and Engineering, Hanyang University Using Subtractive Colors on Film Color absorbing pigments are layered on each other. As white light passes through each layer, different wavelengths are absorbed. The resulting color is produced by subtracting unwanted colors from white. White light Pigment layers Reflecting layer (white paper) M Y C B R G K W GreenRedBlueBlackWhite Cyan YellowMagentaCyan MagentaYellow Black

18. Department of Computer Science and Engineering, Hanyang University 380480580680780 Wavelength (nm) 0 9 Relative power The dashed line represents daylight reflected from sunflower, while the solid line represents the light emitted from the color monitor adjusted to match the color of the sunflower. Metamerism Spectrally different lights that simulate cones identically appear identical. Such colors are called color metamers. This phenomena is called metamerism. Almost all the colors that we see on computer monitors are metamers.

19. Department of Computer Science and Engineering, Hanyang University The Mechanics of Metamerism Under trichromacy, any color stimulus can be matched by a mixture of three primary stimuli. Metamers are colors having the same tristimulus values R, G, and B ; they will match color stimulus C and will appear to be the same color. Wavelength (nm) 780380480580680 0 9 Relative power The two metamers look the same because they have similar tristimulus values. Wavelength (nm) 780380480580680 0 9 Relative power Wavelength (nm) 780380480580680 0 9 Relative power

20. Department of Computer Science and Engineering, Hanyang University Gamut A gamut is the range of colors that a device can render, or detect. The larger the gamut, the more colors can be rendered or detected. A large gamut implies a large color space. 0 0 0.20.40.60.8 0.2 0.4 0.6 0.8 x y Human vision gamut Monitor gamut Photographic film gamut

21. Department of Computer Science and Engineering, Hanyang University Color Spaces A Color Space is a method by which colors are specified, created, and visualized. Colors are usually specified by using three attributes, or coordinates, which represent its position within a specific color space. These coordinates do not tell us what the color looks like, only where it is located within a particular color space. Color models are 3D coordinate systems, and a subspace within that system, where each color is represented by a single point.

22. Department of Computer Science and Engineering, Hanyang University Color Spaces Color Spaces are often geared towards specific applications or hardware. Several types:-  HSI (Hue, Saturation, Intensity) based  RGB (Red, Green, Blue) based  CMY(K) (Cyan, Magenta, Yellow, Black) based  CIE based  Luminance - Chrominance based CIE: International Commission on Illumination

23. Department of Computer Science and Engineering, Hanyang University RGB* One of the simplest color models. Cartesian coordinates for each color; an axis is each assigned to the three primary colors red (R), green (G), and blue (B). Corresponds to the principles of additive colors. Other colors are represented as an additive mix of R, G, and B. Ideal for use in computers. *Red, Green, and Blue Black (0,0,0) Cyan (0,1,1) Green (0,1,0) Yellow (1,1,0) Red (1,0,0) Magenta (1,0,1) Blue (0,0,1) White (1,1,1) RGB Color Space

24. Department of Computer Science and Engineering, Hanyang University RGB Image Data Red Channel Green Channel Full Color Image Blue Channel

25. Department of Computer Science and Engineering, Hanyang University CMY(K)* Main color model used in the printing industry. Related to RGB. Corresponds to the principle of subtractive colors, using the three secondary colors Cyan, Magenta, and Yellow. Theoretically, a uniform mix of cyan, magenta, and yellow produces black (center of picture). In practice, the result is usually a dirty brown-gray tone. So black is often used as a fourth color. *Cyan, Magenta, Yellow, (and blacK) Magenta Yellow Cyan Blue Red Green Black White Producing other colors from subtractive colors.

26. Department of Computer Science and Engineering, Hanyang University CMY Image Data Full Color Image Cyan Image (1-R) Magenta Image (1-G)Yellow Image (1-B)

27. Department of Computer Science and Engineering, Hanyang University CMY – RBG Transformation The following matrices will perform transformations between RGB and CMY color spaces. Note that:-  R = Red  G = Green  B = Blue  C = Cyan  M = Magenta  Y = Yellow  All values for R, G, B and C, M, Y must first be normalized.

28. Department of Computer Science and Engineering, Hanyang University HSI / HSL / HSV* Very similar to the way human visions see color. Works well for natural illumination, where hue changes with brightness. Used in machine color vision to identify the color of different objects. Image processing applications like histogram operations, intensity transformations, and convolutions operate on only an image's intensity and are performed much easier on an image in the HSI color space. *H=Hue, S = Saturation, I (Intensity) = B (Brightness), L = Lightness, V = Value

29. Department of Computer Science and Engineering, Hanyang University HSI Color Space Hue  What we describe as the color of the object.  Hues based on RGB color space.  The hue of a color is defined by its counterclockwise angle from Red (0°); e.g. Green = 120 °, Blue = 240 °. RGB Color Space RGB cube viewed from gray-scale axis RGB cube viewed from gray-scale axis, and rotated 30° HSI Color Wheel Red 0º Gre en 120º Blue 240º  Saturation  Degree to which hue differs from neutral gray.  100% = Fully saturated, high contrast between other colors.  0% = Shade of gray, low contrast.  Measured radially from intensity axis. 0%0% Saturation 100%

30. Department of Computer Science and Engineering, Hanyang University HSI Color Space Intensity  Brightness of each Hue, defined by its height along the vertical axis.  Max saturation at 50% Intensity.  As Intensity increases or decreases from 50%, Saturation decreases.  Mimics the eye response in nature; As things become brighter they look more pastel until they become washed out.  Pure white at 100% Intensity. Hue and Saturation undefined.  Pure black at 0% Intensity. Hue and Saturation undefined. Hue Saturation0%100% Intensity 100% 0%

31. Department of Computer Science and Engineering, Hanyang University HSI Image Data Hue Channel Saturation Channel Intensity Channel Full Image

32. Department of Computer Science and Engineering, Hanyang University CIE L*a*b* Color Space / CIELAB Second of two systems adopted by CIE in 1976 as models that better showed uniform color spacing in their values. Based on the earlier (1942) color opposition system by Richard Hunter called L, a, b. color opposition system Very important for desktop color. Basic color model in Adobe PostScript (level 2 and level 3) Used for color management as the device independent model of the ICC* device profiles. CIE L*a*b* color axes *International Color Consortium

33. Department of Computer Science and Engineering, Hanyang University CIE L*a*b* (cont’d) Central vertical axis : Lightness (L*), runs from 0 (black) to 100 (white). a-a' axis: +a values indicate amounts of red, -a values indicate amounts of green. b-b' axis, +b indicates amounts of yellow; -b values indicates amounts of blue. For both axes, zero is neutral gray. Only values for two color axes (a*, b*) and the lightness or grayscale axis (L*) are required to specify a color. CIELAB Color difference,  E* ab, is between two points is given by: +a+a -a-a -b-b +b+b 100 0 L* CIE L*a*b* color axes (L 1 *, a 1 *, b 1 *) (L 2 *, a 2 *, b 2 *)

34. Department of Computer Science and Engineering, Hanyang University CIELAB Image Data Full Color Image L data L-a channelL-b channel

35. Department of Computer Science and Engineering, Hanyang University Scene Radiance L Lens Image Irradiance E Camera Electronics Scene Image Irradiance E Measured Pixel Values, I Non-linear Mapping! Linear Mapping! Before light hits the image plane: After light hits the image plane: Can we go from measured pixel value, I, to scene radiance, L? Relationship between Scene and Image Brightness

36. Department of Computer Science and Engineering, Hanyang University Demosaicking Cf. 3CCD camera

37. Department of Computer Science and Engineering, Hanyang University The camera response function relates image irradiance at the image plane to the measured pixel intensity values. Camera Electronics Image Irradiance E Measured Pixel Values, I (Grossberg and Nayar) Relation between Pixel Values I and Image Irradiance E

38. Department of Computer Science and Engineering, Hanyang University Important preprocessing step for many vision and graphics algorithms such as photometric stereo, invariants, de-weathering, inverse rendering, image based rendering, etc. Use a color chart with precisely known reflectances. Irradiance = const * Reflectance Pixel Values 3.1%9.0%19.8%36.2%59.1%90% Use more camera exposures to fill up the curve. Method assumes constant lighting on all patches and works best when source is far away (example sunlight). Unique inverse exists because g is monotonic and smooth for all cameras. 0 255 01 g ? ? Radiometric Calibration

39. Department of Computer Science and Engineering, Hanyang University Dynamic Range

40. Department of Computer Science and Engineering, Hanyang University Dynamic Range: Range of brightness values measurable with a camera (Hood 1986) High Exposure Image Low Exposure Image We need 5-10 million values to store all brightnesses around us. But, typical 8-bit cameras provide only 256 values!! Today’s Cameras: Limited Dynamic Range Today’s Cameras: Limited Dynamic Range The Problem of Dynamic Range

41. Department of Computer Science and Engineering, Hanyang University High dynamic range imaging Techniques  Debevec: http://www.debevec.org/Research/HDR/ http://www.debevec.org/Research/HDR/  Columbia: http://www.cs.columbia.edu/CAVE/tomoo/RRHomePage/rrgallery.html http://www.cs.columbia.edu/CAVE/tomoo/RRHomePage/rrgallery.html

42. Department of Computer Science and Engineering, Hanyang University Color Discrimination Active approach  Using controlled lights Passive approach  Using optical filters camera LED Cluster controller scene illumination 1Illumination 2

43. Department of Computer Science and Engineering, Hanyang University Visual effect of illumination Camera Blue Channel Camera Green Channel Camera Red Channel Synthetic Illumination L A Halogen Lamp RGB Distance: 115.86 RGB Distance: 98.12 RGB Distance: 92.85 Xenon Lamp

44. Department of Computer Science and Engineering, Hanyang University Optimal illumination

45. Department of Computer Science and Engineering, Hanyang University Imaging for Autonomous Vehicle For traffic lights  Passive approach  Using optimized color filters For pedestrian detection  Multispectral/hyperspectral imaging  Infrared band

46. Department of Computer Science and Engineering, Hanyang University Segmentation Keying Interactive segmentation [ 서울대 ]

47. Department of Computer Science and Engineering, Hanyang University Virtual Studio NHK STRL: Synthevision, VS, DTPP (1989~1992) VS Overview paper: S.Gibbs et al.(1996)