1. Colors and sensors Slides from Bill Freeman, Fredo Durand, Rob Fergus, and David Forsyth, Alyosha Efros

2. Agenda Project 1 delayed till Thursday October 9 th Color Sensors Matlab intro

3. Project 1: Demosaicing Warning; it might take some time –Getting familiar with Matlab –Writeup solutions in html and submit to EEE –I put up a project template (with sample code & writeup)

4. Image Formation Digital Camera The Eye Film

5. Digital camera A digital camera replaces film with a sensor array –Each cell in the array is light-sensitive diode that converts photons to electrons –Two common types Charge Coupled Device (CCD) CMOS –http://electronics.howstuffworks.com/digital-camera.htmhttp://electronics.howstuffworks.com/digital-camera.htm Slide by Steve Seitz

6. Sensor Array CMOS sensor

7. Sampling and quantizing brightness

8. The real world of colored light: why is color useful? Find things to eat Spot dangerous things

9. What’s the physics behind color?

10. The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm. © Stephen E. Palmer, 2002

11. The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002

12. The Physics of Light Some examples of the reflectance spectra of surfaces Wavelength (nm) % Photons Reflected Red 400 700 Yellow 400 700 Blue 400 700 Purple 400 700 © Stephen E. Palmer, 2002

13. Electromagnetic Spectrum http://www.yorku.ca/eye/photopik.htm Human Luminance Sensitivity Function

14. Computer Vision - A Modern Approach Set: Color Slides by D.A. Forsyth Black body radiators Construct a hot body with near-zero albedo (black body) –Easiest way to do this is to build a hollow metal object with a tiny hole in it, and look at the hole. The spectral power distribution of light leaving this object is a simple function of temperature This leads to the notion of “color temperature” --- the temperature of a black body that would look the same

15. Why do we see light of these wavelengths? © Stephen E. Palmer, 2002 …because that’s where the Sun radiates EM energy Visible Light Plank’s law for Blackbody radiation Surface of the sun: ~5800K

16. The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions area mean variance © Stephen E. Palmer, 2002

17. The Psychophysical Correspondence MeanHue # Photons Wavelength © Stephen E. Palmer, 2002

18. The Psychophysical Correspondence VarianceSaturation Wavelength # Photons © Stephen E. Palmer, 2002

19. The Psychophysical Correspondence AreaBrightness # Photons Wavelength © Stephen E. Palmer, 2002

20. Spectral Image Formation I(λ) R(λ) Si(λ)Si(λ) I(λ) R(λ) From Foundation of Vision by Brian Wandell, Sinauer Associates, 1995

21. Spectral Image Formation I(λ) – Illumination Spectrum R(λ)- Surface reflectance/transmission S i (λ) - Spectral sensitivity of photoreceptor i Pixel value / Perceived color depends on all 3 terms!  Problem of color constancy P i (λ) = I(λ)R(λ)S i (λ)

22. Color appearance depends on nearby colors Top pink should look stronger

23. Color names for cartoon spectra 400 500 600 700 nm red green blue 400 500 600 700 nm cyan magenta yellow 400 500 600 700 nm Slide credit: W. Freeman

24. Additive color mixing 400 500 600 700 nm red green Red and green make… 400 500 600 700 nm yellow Yellow! When colors combine by adding the color spectra. Example color displays that follow this mixing rule: CRT phosphors, multiple projectors aimed at a screen, Polachrome slide film. Slide credit: W. Freeman

25. Additive color mixing of illuminants

26. Subtractive color mixing When colors combine by multiplying the color spectra. Examples that follow this mixing rule: most photographic films, paint, cascaded optical filters, crayons. 400 500 600 700 nm cyan yellow 400 500 600 700 nm Cyan and yellow (in crayons, called “blue” and yellow) make… 400 500 600 700 nm Green! green Slide credit: W. Freeman

27. Subtractive color mixing of materials Light reflecting off colored object E.g. printing inks Wikipedia

28. Color matching experiment Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

29. Color matching experiment 1

30. p 1 p 2 p 3

31. Color matching experiment 1 p 1 p 2 p 3

32. Color matching experiment 1 p 1 p 2 p 3 The primary color amounts needed for a match

33. Color matching experiment 2

34. p 1 p 2 p 3

35. Color matching experiment 2 p 1 p 2 p 3

36. Color matching experiment 2 p 1 p 2 p 3 We say a “negative” amount of p 2 was needed to make the match, because we added it to the test color’s side. The primary color amounts needed for a match: p 1 p 2 p 3

37. Measure color by color-matching paradigm Pick a set of 3 primary color lights. Find the amounts of each primary, e 1, e 2, e 3, needed to match some spectral signal, t. Those amounts, e 1, e 2, e 3, describe the color of t. If you have some other spectral signal, s, and s matches t perceptually, then e 1, e 2, e 3 will also match s, by Grassman’s laws. Why this is useful—it lets us: –Predict the color of a new spectral signal –Translate to representations using other primary lights.

38. Goal: compute the color match for any color signal for any set of primary colors Examples of why you’d want to do that: –Want to paint a carton of Kodak film with the Kodak yellow color. –Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color. –Want the colors in the world, on a monitor, and in a print format to all look the same.

39. Color matching functions for a particular set of monochromatic primaries p 1 = 645.2 nm p 2 = 525.3 nm p 3 = 444.4 nm Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

40. Questions?

41. Some other color spaces…

42. NTSC color components: Y, I, Q

43. NTSC - RGB

44. HSV hexcone Forsyth & Ponce

45. Hue Saturation Value Value: from black to white Hue: dominant color (red, orange, etc) Saturation: from gray to vivid color HSV double cone value saturation hue

46. CCD color sampling

48. The eye’s approach to color imaging

49. © Stephen E. Palmer, 2002 Cones cone-shaped less sensitive operate in high light color vision Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision

50. Human eye photoreceptor spectral sensitivities Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 What colors would these look like?

51. Color Sensing in Camera (RGB) 3-chip vs. 1-chip: quality vs. cost Why more green? http://www.coolditionary.com/words/Bayer-filter.wikipedia http://www.cooldictionary.com/words/Bayer-filter.wikipedia Why 3 colors? Slide by Steve Seitz

56. CCD color filter pattern detector

57. The cause of color moire detector Fine black and white detail in image mis-interpreted as color information.

59. Typical color moire patterns Blow-up of electronic camera image. Notice spurious colors in the regions of fine detail in the plants.

60. Color sampling artifacts

61. Human receptors vs CCD sensors Distribution of incoming luminance into CCD sensors

62. Gamma correction Iout = Iin^(gamma), where gamma < 1