Colors and sensors Slides from Bill Freeman, Fredo Durand, Rob Fergus, and David Forsyth, Alyosha Efros

Agenda Project 1 delayed till Thursday October 9 th Color Sensors Matlab intro

Project 1: Demosaicing Warning; it might take some time –Getting familiar with Matlab –Writeup solutions in html and submit to EEE –I put up a project template (with sample code & writeup)

Image Formation Digital Camera The Eye Film

Digital camera A digital camera replaces film with a sensor array –Each cell in the array is light-sensitive diode that converts photons to electrons –Two common types Charge Coupled Device (CCD) CMOS – Slide by Steve Seitz

Sensor Array CMOS sensor

Sampling and quantizing brightness

The real world of colored light: why is color useful? Find things to eat Spot dangerous things

What’s the physics behind color?

The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength nm. © Stephen E. Palmer, 2002

The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002

The Physics of Light Some examples of the reflectance spectra of surfaces Wavelength (nm) % Photons Reflected Red Yellow Blue Purple © Stephen E. Palmer, 2002

Electromagnetic Spectrum Human Luminance Sensitivity Function

Computer Vision - A Modern Approach Set: Color Slides by D.A. Forsyth Black body radiators Construct a hot body with near-zero albedo (black body) –Easiest way to do this is to build a hollow metal object with a tiny hole in it, and look at the hole. The spectral power distribution of light leaving this object is a simple function of temperature This leads to the notion of “color temperature” --- the temperature of a black body that would look the same

Why do we see light of these wavelengths? © Stephen E. Palmer, 2002 …because that’s where the Sun radiates EM energy Visible Light Plank’s law for Blackbody radiation Surface of the sun: ~5800K

The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions area mean variance © Stephen E. Palmer, 2002

The Psychophysical Correspondence MeanHue # Photons Wavelength © Stephen E. Palmer, 2002

The Psychophysical Correspondence VarianceSaturation Wavelength # Photons © Stephen E. Palmer, 2002

The Psychophysical Correspondence AreaBrightness # Photons Wavelength © Stephen E. Palmer, 2002

Spectral Image Formation I(λ) R(λ) Si(λ)Si(λ) I(λ) R(λ) From Foundation of Vision by Brian Wandell, Sinauer Associates, 1995

Spectral Image Formation I(λ) – Illumination Spectrum R(λ)- Surface reflectance/transmission S i (λ) - Spectral sensitivity of photoreceptor i Pixel value / Perceived color depends on all 3 terms!  Problem of color constancy P i (λ) = I(λ)R(λ)S i (λ)

Color appearance depends on nearby colors Top pink should look stronger

Color names for cartoon spectra nm red green blue nm cyan magenta yellow nm Slide credit: W. Freeman

Additive color mixing nm red green Red and green make… nm yellow Yellow! When colors combine by adding the color spectra. Example color displays that follow this mixing rule: CRT phosphors, multiple projectors aimed at a screen, Polachrome slide film. Slide credit: W. Freeman

Additive color mixing of illuminants

Subtractive color mixing When colors combine by multiplying the color spectra. Examples that follow this mixing rule: most photographic films, paint, cascaded optical filters, crayons nm cyan yellow nm Cyan and yellow (in crayons, called “blue” and yellow) make… nm Green! green Slide credit: W. Freeman

Subtractive color mixing of materials Light reflecting off colored object E.g. printing inks Wikipedia

Color matching experiment Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

Color matching experiment 1

p 1 p 2 p 3

Color matching experiment 1 p 1 p 2 p 3

Color matching experiment 1 p 1 p 2 p 3 The primary color amounts needed for a match

Color matching experiment 2

p 1 p 2 p 3

Color matching experiment 2 p 1 p 2 p 3

Color matching experiment 2 p 1 p 2 p 3 We say a “negative” amount of p 2 was needed to make the match, because we added it to the test color’s side. The primary color amounts needed for a match: p 1 p 2 p 3

Measure color by color-matching paradigm Pick a set of 3 primary color lights. Find the amounts of each primary, e 1, e 2, e 3, needed to match some spectral signal, t. Those amounts, e 1, e 2, e 3, describe the color of t. If you have some other spectral signal, s, and s matches t perceptually, then e 1, e 2, e 3 will also match s, by Grassman’s laws. Why this is useful—it lets us: –Predict the color of a new spectral signal –Translate to representations using other primary lights.

Goal: compute the color match for any color signal for any set of primary colors Examples of why you’d want to do that: –Want to paint a carton of Kodak film with the Kodak yellow color. –Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color. –Want the colors in the world, on a monitor, and in a print format to all look the same.

Color matching functions for a particular set of monochromatic primaries p 1 = nm p 2 = nm p 3 = nm Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

Questions?

Some other color spaces…

NTSC color components: Y, I, Q

NTSC - RGB

HSV hexcone Forsyth & Ponce

Hue Saturation Value Value: from black to white Hue: dominant color (red, orange, etc) Saturation: from gray to vivid color HSV double cone value saturation hue

CCD color sampling

The eye’s approach to color imaging

© Stephen E. Palmer, 2002 Cones cone-shaped less sensitive operate in high light color vision Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision

Human eye photoreceptor spectral sensitivities Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 What colors would these look like?

Color Sensing in Camera (RGB) 3-chip vs. 1-chip: quality vs. cost Why more green? Why 3 colors? Slide by Steve Seitz

CCD color filter pattern detector

The cause of color moire detector Fine black and white detail in image mis-interpreted as color information.

Typical color moire patterns Blow-up of electronic camera image. Notice spurious colors in the regions of fine detail in the plants.

Color sampling artifacts

Human receptors vs CCD sensors Distribution of incoming luminance into CCD sensors

Gamma correction Iout = Iin^(gamma), where gamma < 1