Photometric Image Formation CSE 559: Computer Vision Guest Lecturer: Austin Abrams Images/Demo from Steve Seitz, Wikipedia

How are images made? One half: geometric vision “how the pixel projected onto the image” Today: photometric vision (aka radiometric) “how the pixel got its color”

Vision and Graphics Image Properties of a scene Computer Graphics

Image Formation Approach Come up with a model for how the scene was created Given images, find the most likely properties that fit that model

Diffuse Surfaces Brightness of a pixel depends on: object color lighting direction surface normal But NOT view direction!

Lambertian Cosine Law L N The intensity of an observed diffuse object is proportional to the cosine of the angle between the normal and lighting direction I = ρ cos θ L N θ = ρ |L||N| cos θ = ρ L  N

=  L  N = L N

= x I = ρ L N

Recovering Albedo and Normals Can you decompose a single image into its albedo and normal images?

x = x x

Photometric Stereo Given multiple images taken with varying illumination, recover albedo and normals. take pictures in dark room with varying illumination. estimate lighting directions L. recover albedo and normals.

Side note 1: How to get the lighting direction? Put a shiny sphere in the scene Sphere’s geometry (normals) are known Find specular highlight

Side-note 2: Why “Stereo”? Surface normals provide constraints on depth differences

Photometric Stereo If L is known, and albedo is grayscale this is a linear problem. I = ρ(L  N) = ρ (Lx Nx + Ly Ny + Lz Nz ) = Lx Nxρ + Ly Nyρ + Lz Nzρ = Lx a + Ly b + Lz c

For each pixel: I = ρ(L  N) = Lx a + Ly b + Lz c I1 Lx1 Ly1 Lz1 I2 … Lxn Lyn Lzn I1 I2 I3 In a b c = Then: ρ = sqrt(a2 + b2 + c2) N = (a,b,c) / ρ

Demo

When does this model fail? I ≠ ρ (L  N)

I = ρ max(L  N, 0) Attached shadows L  N = 0 L  N > 0

Cast Shadows, Ambient Light I = ρ (S L  N + a) S = 0 or 1

Radiometric Camera Calibration Pixel intensities are usually not proportional to the energy that hit the CCD RAW image Published image

Radiometric Camera Calibration Published f RAW

Radiometric Camera Calibration Observed = f(RAW) (Grossberg and Nayar) f -1 (Observed) = RAW

Radiometric Camera Calibration How do you model f -1? f -1(x) = xγ f -1(x) = c0 + c1x + c2x2 + c3x3 + … f -1(x) = f0(x) + f1(x) c1 + f2(x)c2 + … mean camera curve basis camera curves

Radiometric Camera Calibration I = f (ρ (S L  N + a)) Adding exposure: I = f (e ρ (S L  N + a))

Heliometric Stereo I = f (e ρ (S L  N + a)) Given lots of images from a stable webcam, use lighting from the sun to recover: I = f (e ρ (S L  N + a))

Heliometric Stereo

Heliometric Stereo

Heliometric Stereo

Radiometric Camera Calibration If you can control the exposure… Take two pictures with different known exposures (e.g. 0.5 second and 1 second): Observed2 = f (e2 RAW) Observed1 = f (e1 RAW) f -1 (Observed1) = e1 RAW f -1 (Observed2) = e2 RAW f -1 (Observed1) e1 f -1 (Observed2) e2 = Solve for the best f -1 that fits your model

Heliometric Stereo I = f (e ρ (S L  N + a)) The following should hold for each pixel in each image: I = f (e ρ (S L  N + a)) f : the camera’s response curve e: that image’s exposure value a: that image’s ambient light S: 0 if that pixel is in shadow at that time, 1 otherwise N: that pixel’s surface normal ρ: that pixel’s albedo

Heliometric Stereo I = f (e ρ (S L  N + a)) Step 1: pixel-level thresholding to find shadows Step 2: initialize all variables I = f (e ρ (S L  N + a))

Heliometric Stereo I = f (e ρ (S L  N + a)) I = f (e ρ (S L  N + a)) Step 3: fix f, e, and a, solve for ρ and N. Step 4: fix ρ and N, solve for f, e, and a. Step 5: goto 3. I = f (e ρ (S L  N + a)) I = f (e ρ (S L  N + a))

The life of a photon

BRDF Bi-Directional Reflectance Distribution Function given incoming and outgoing rays, what proportion of light is reflected? just a big word that describes how light’s energy is transferred upon hitting a surface

BRDF Almost nobody actually tries to model a full BRDF. Why? Build a lighting model with fewer parameters that approximate the BRDF Diffuse lighting model is very common

Diffuse Surfaces