1. High Dynamic Range from Multiple Images: Which Exposures to Combine?
Michael Grossberg and Shree Nayar
CAVE Lab, Columbia University
ICCV Workshop on CPMCV
October, 2003, Nice, France
Partially funded by NSF ITR Award, DARPA/ONR MURI
Work on Imaging Models
Joint with Shree Nayar in the CAVE vision lab at Columbia
Partially funded by the NSF and DARPA

2. Cameras Have Limited Dynamic Range
Short Exposure
Long Exposure
Desired Image

3. Combining Different Exposures+
Low Dynamic Range Exposures
Combination Yields High Dynamic Range
[Ginosar and Zeevi, 88, Madden, 93,
Mann and Picard, 95, Debevec and Malik, 97,
Mitsunaga and Nayar, 99]

4. Combining Different Exposures+
Low Dynamic Range Exposures
Combination Yields High Dynamic Range
[Ginosar and Zeevi, 88, Madden, 93,
Mann and Picard, 95, Debevec and Malik, 97,
Mitsunaga and Nayar, 99]

5. (Optical Attenuation)
The Camera Response
255
Camera Response
Image
Intensity B f
Scene
Radiance
Linear Function
(Optical Attenuation)
Image
Irradiance L s E

6. From Response To Measured Irradiance Levels
Response Function f
Brightness Levels B
Irradiance E
Measured Irradiance Levels

7. Where do you want your bits?
Response Function f
Coarse
quantization
Brightness B
High Dynamic
Range
Irradiance E
Response Function f
Fine
quantization
Brightness B
Low Dynamic
Range
Irradiance E

8. Effective Camera from Multiple Exposures
Goal
Acquired Images
Image from Effective Camera
Capture
High
Dynamic
Range +
Capture
Irradiance
Uniformly +

9. Flexible Dynamic Range Imaging:
Can we create an effective camera with a desired response?
How many exposures are needed?
Which exposures to acquire?
How to combine the acquired images?

10. Irradiance Levels From Multiple Exposures
f(E) 2 1 3
Exposure e1 = 1
Sum
Irradiance E E1 E2 E3
f(e2E)
Exposure e2 2 1 3
E3 / e3
E2 / e2
E1 / e1
h(E) 2 1 3 6 4 5
Effective Camera E1 ^ E6 E5 E4 E3 E2

11. Response of the Effective Camera
Effective Response
Number of exposures
Camera Response
Exposures
Irradiance
Theorem: The sum of a set of images of a scene taken at different exposures includes all the information in the individual exposures.

12. Camera Response Emulationh ~
Emulated Response
depends on: h
f ,
e = (e1, … ,en)
Brightness Levels B g
Desired Response
How can we tell if h emulates g well?
Naïve answer: | h – g | < e
Irradiance E
Level spacing characterizes similarity

13. How Response Determines Level Spacing
Observation:
The derivative determines the distances between levels.
Larger
Derivative
Brightness Levels B
Sparse
Spacing
Smaller
Derivatives
Dense
Spacing
Irradiance E
Measured Irradiance Levels

14. The Objective Function
Spacing Based Comparison:
Number of Exposures
Exposure Values
Desired Response
Effective Response
Weight
Weight prevents penalizing success: 0, 1,
otherwise
Brightness B g
Irradiance E h
w=0
w=1

15. Accounting for Camera Noise
Add term to Penalize Variance
Example: Gaussian Noise
Full Noise model depends on
Thermal, Shot, or Read Noise
Integration Time, Gain, Aperture
Camera Response
Weight
Noise Term
Noise Variance
Exposure Values

16. Which Exposures and How Many?
For fixed n, find minimizing exposure values e
Choose min n such that error within tolerance
Method: Exhaustive search
Objective function not continuous
Only need to search actual settings
Offline build table of exposures

17. Flexible Dynamic Range Imaging
Linear
Gamma =
1/2
Constant
contrast
(log) 1 2
1.003
1.015
3.094
20.24
9.91
4.689
4.831
3.979 3
2.985
1.019
1.007
5.146
3.019
1.006
88.38
37.23
29.18
16.01 4
3.672
1.166
1.031
11.23
5.049
2.866
280
144.9
64.22 5
4.975
1.078
18.56
8.564
763.5
305.4 6
5.636
33.7
1130
Effective
Camera
Number of
Exposures
Exposure Values

18. Flexible Dynamic Range Imaging
Linear
Gamma =
1/2
Constant
contrast
(log) 1 2
1.003
1.015
3.094
20.24
9.91
4.689
4.831
3.979 3
2.985
1.019
1.007
5.146
3.019
1.006
88.38
37.23
29.18
16.01 4
3.672
1.166
1.031
11.23
5.049
2.866
280
144.9
64.22 5
4.975
1.078
18.56
8.564
763.5
305.4 6
5.636
33.7
1130
Effective
Camera
Number of
Exposures
Exposure Values
Desired Response
Constant Contrast
Gamma 1/2
Linear
Desired Response
Constant Contrast
Gamma 1/2
Linear
High :16,000
Low :256
Dynamic Range
Medium 1:1,000
High :16,000
Low :256
Dynamic Range

19. Baseline Exposure Values
Typically exposures are doubled
Baseline:
Combine the exposures e =(1,2,4)
[Ginosar and Zeevi, 88, Madden, 93,
Mann and Picard, 95, Debevec and Malik, 97,
Mitsunaga and Nayar, 99]

20. Increased Dynamic Range Linear Camera
Real Camera: f linear
Desired Camera: g linear (greater dynamic range)
Brightness
Brightness
Desired Response
0.8
0.6
0.4
0.2
0.0
1.0
Irradiance E
Brightness B
4-bit real camera
8-bit real camera
0.8
0.6
0.4
0.2
0.0
1.0
Desired Response
Irradiance E
Brightness B
Baseline Response (1,2,4)
Computed Response
(1, 1.05, 1.11)
Baseline Response (1,2,4)
Computed Response
(1, 1.003, 2.985)

21. Linear Camera: Synthetic Ramp Image
High Dynamic Range Linear Camera
From baseline exposures (1,2,4)
From computed exposures (1,1.05,1.11)

22. Linear Camera: Synthetic Ramp Image
High Dynamic Range Linear Camera
From baseline exposures (1,2,4)
From computed exposures (1,1.05,1.11)
As above with noise

23. Linear Camera: Image of Cloth
Ground Truth (HDR image)
Baseline Exposures (1,2,4)
Computed Exposures (1,1.05,1.11)

24. Constant Contrast from Linear Cameras
Real Camera: f linear
Desired Camera: g log response (constant contrast)
0.8
0.6
0.4
0.2
0.0
1.0
Irradiance E
Brightness B
Desired
Response
3 Exposures
Irradiance E
1.0
0.8
0.6
0.4
0.2
0.0
Brightness B
Desired
Response
5 Exposures
Baseline Response
(1,2,4)
Computed Response
(1,9.91,88.38)
Computed
Response
Baseline Response

25. Constant Contrast : Image of Tiles
Baseline Exposures
Computed Exposures

26. Linear Camera from Non-linear Camera
Real Camera: f non-linear (Nikon 990)
Desired Camera: g linear
Baseline Input Exposures
Iso-brightness
Computed Input Exposures
Combined
Brightness
Brightness
Iso-brightness
Combined

27. Summary Combine images using summation
Method finds number of exposures and exposure values to use
Emulation of a variety of cameras:
Flexible Dynamic Range Imaging

28. Colors