High Dynamic Range from Multiple Images: Which Exposures to Combine? Michael Grossberg and Shree Nayar CAVE Lab, Columbia University ICCV Workshop on CPMCV October, 2003, Nice, France Partially funded by NSF ITR Award, DARPA/ONR MURI Work on Imaging Models Joint with Shree Nayar in the CAVE vision lab at Columbia Partially funded by the NSF and DARPA

Cameras Have Limited Dynamic Range Short Exposure Long Exposure Desired Image

Combining Different Exposures + Low Dynamic Range Exposures Combination Yields High Dynamic Range [Ginosar and Zeevi, 88, Madden, 93, Mann and Picard, 95, Debevec and Malik, 97, Mitsunaga and Nayar, 99]

Combining Different Exposures + Low Dynamic Range Exposures Combination Yields High Dynamic Range [Ginosar and Zeevi, 88, Madden, 93, Mann and Picard, 95, Debevec and Malik, 97, Mitsunaga and Nayar, 99]

(Optical Attenuation) The Camera Response 255 Camera Response Image Intensity B f Scene Radiance Linear Function (Optical Attenuation) Image Irradiance L s E

From Response To Measured Irradiance Levels Response Function f Brightness Levels B Irradiance E Measured Irradiance Levels

Where do you want your bits? Response Function f Coarse quantization Brightness B High Dynamic Range Irradiance E Response Function f Fine quantization Brightness B Low Dynamic Range Irradiance E

Effective Camera from Multiple Exposures Goal Acquired Images Image from Effective Camera Capture High Dynamic Range + Capture Irradiance Uniformly +

Flexible Dynamic Range Imaging: Can we create an effective camera with a desired response? How many exposures are needed? Which exposures to acquire? How to combine the acquired images?

Irradiance Levels From Multiple Exposures f(E) 2 1 3 Exposure e1 = 1 Sum Irradiance E E1 E2 E3 f(e2E) Exposure e2 2 1 3 E3 / e3 E2 / e2 E1 / e1 h(E) 2 1 3 6 4 5 Effective Camera E1 ^ E6 E5 E4 E3 E2

Response of the Effective Camera Effective Response Number of exposures Camera Response Exposures Irradiance Theorem: The sum of a set of images of a scene taken at different exposures includes all the information in the individual exposures.

Camera Response Emulation h ~ Emulated Response depends on: h f , e = (e1, … ,en) Brightness Levels B g Desired Response How can we tell if h emulates g well? Naïve answer: | h – g | < e Irradiance E Level spacing characterizes similarity

How Response Determines Level Spacing Observation: The derivative determines the distances between levels. Larger Derivative Brightness Levels B Sparse Spacing Smaller Derivatives Dense Spacing Irradiance E Measured Irradiance Levels

The Objective Function Spacing Based Comparison: Number of Exposures Exposure Values Desired Response Effective Response Weight Weight prevents penalizing success: 0, 1, otherwise Brightness B g Irradiance E h w=0 w=1

Accounting for Camera Noise Add term to Penalize Variance Example: Gaussian Noise Full Noise model depends on Thermal, Shot, or Read Noise Integration Time, Gain, Aperture Camera Response Weight Noise Term Noise Variance Exposure Values

Which Exposures and How Many? For fixed n, find minimizing exposure values e Choose min n such that error within tolerance Method: Exhaustive search Objective function not continuous Only need to search actual settings Offline build table of exposures

Flexible Dynamic Range Imaging Linear Gamma = 1/2 Constant contrast (log) 1 2 1.003 1.015 3.094 20.24 9.91 4.689 4.831 3.979 3 2.985 1.019 1.007 5.146 3.019 1.006 88.38 37.23 29.18 16.01 4 3.672 1.166 1.031 11.23 5.049 2.866 280 144.9 64.22 5 4.975 1.078 18.56 8.564 763.5 305.4 6 5.636 33.7 1130 Effective Camera Number of Exposures Exposure Values

Flexible Dynamic Range Imaging Linear Gamma = 1/2 Constant contrast (log) 1 2 1.003 1.015 3.094 20.24 9.91 4.689 4.831 3.979 3 2.985 1.019 1.007 5.146 3.019 1.006 88.38 37.23 29.18 16.01 4 3.672 1.166 1.031 11.23 5.049 2.866 280 144.9 64.22 5 4.975 1.078 18.56 8.564 763.5 305.4 6 5.636 33.7 1130 Effective Camera Number of Exposures Exposure Values Desired Response Constant Contrast Gamma 1/2 Linear Desired Response Constant Contrast Gamma 1/2 Linear High 1:16,000 Low 1:256 Dynamic Range Medium 1:1,000 High 1:16,000 Low 1:256 Dynamic Range

Baseline Exposure Values Typically exposures are doubled Baseline: Combine the exposures e =(1,2,4) [Ginosar and Zeevi, 88, Madden, 93, Mann and Picard, 95, Debevec and Malik, 97, Mitsunaga and Nayar, 99]

Increased Dynamic Range Linear Camera Real Camera: f linear Desired Camera: g linear (greater dynamic range) Brightness Brightness Desired Response 0.8 0.6 0.4 0.2 0.0 1.0 Irradiance E Brightness B 4-bit real camera 8-bit real camera 0.8 0.6 0.4 0.2 0.0 1.0 Desired Response Irradiance E Brightness B Baseline Response (1,2,4) Computed Response (1, 1.05, 1.11) Baseline Response (1,2,4) Computed Response (1, 1.003, 2.985)

Linear Camera: Synthetic Ramp Image High Dynamic Range Linear Camera From baseline exposures (1,2,4) From computed exposures (1,1.05,1.11)

Linear Camera: Synthetic Ramp Image High Dynamic Range Linear Camera From baseline exposures (1,2,4) From computed exposures (1,1.05,1.11) As above with noise

Linear Camera: Image of Cloth Ground Truth (HDR image) Baseline Exposures (1,2,4) Computed Exposures (1,1.05,1.11)

Constant Contrast from Linear Cameras Real Camera: f linear Desired Camera: g log response (constant contrast) 0.8 0.6 0.4 0.2 0.0 1.0 Irradiance E Brightness B Desired Response 3 Exposures Irradiance E 1.0 0.8 0.6 0.4 0.2 0.0 Brightness B Desired Response 5 Exposures Baseline Response (1,2,4) Computed Response (1,9.91,88.38) Computed Response Baseline Response

Constant Contrast : Image of Tiles Baseline Exposures Computed Exposures

Linear Camera from Non-linear Camera Real Camera: f non-linear (Nikon 990) Desired Camera: g linear Baseline Input Exposures Iso-brightness Computed Input Exposures Combined Brightness Brightness Iso-brightness Combined

Summary Combine images using summation Method finds number of exposures and exposure values to use Emulation of a variety of cameras: Flexible Dynamic Range Imaging

Colors