1. EE 7700 High Dynamic Range Imaging

2. Bahadir K. Gunturk2 References Slides and papers by Debevec, Ward, Pattaniak, Nayar, Durand, et al… http://people.csail.mit.edu/fredo/PUBLI/Siggraph2002/

3. Bahadir K. Gunturk3 High Dynamic Range (HDR) Imaging star light moon light office light day light search light 10 -6 10 -2 10 1 10 2 10 4 10 8 10 0 The range of luminances is more than 10^14 candela/m2 Range of human eye at an instant is around 10^4:1 (4log units) Human eye can adapt to see much wider range. Candela (cd) is the unit of luminous intensity (power emitted by a light source in a particular direction, with wavelengths weighted by the sensitivity of the human eye.) A common candle emits roughly 1 cd.candle A 100 W incandescent lightbulb emits about 120 cd.lightbulb

4. Bahadir K. Gunturk4 Spectral Sensitivity of Human Visual System: Luminosity Function Photopic (black) and scotopic [1] (green) luminosity functions. The photopic includes the CIE 1931 standard [2] (solid), the Judd-Vos 1978 modified data [3] (dashed), and the Sharpe, Stockman, Jagla & Jägle 2005 data [4] (dotted). The horizontal axis is wavelength in nm. (from Wikipedia) One candela is defined as the luminous intensity of a monochromatic 540 THz light source that has a radiant intensity of 1/683 watts per steradian, or about 1.464 mW/sr. The 540 THz frequency corresponds to a wavelength of about 555 nm, which is green light near the peak of the eye's response. A typical candle produces very roughly one candela of luminous intensity. Quantity Derived SI Unit Symbol Luminance candela per square meter cd/m2 Luminous flux lumen cd * sr = lm Illuminance lux lm/m2 = lx

5. Bahadir K. Gunturk5 HDR star light moon light office light day light search light 10 -6 10 -2 10 1 10 2 10 4 10 8 10 0 The range of radiances is more than 10^14 candela/m2 0 255 Range of Typical Displays: from ~1 to ~100 cd/m 2

6. Bahadir K. Gunturk6 1000 cd/m^2 Cone dominated log L a Gain rod cone log Gain 0 24 6-2 -4 -6 Sensitivity of Eye

7. Bahadir K. Gunturk7 0.04 cd/m^2 Rod dominated log L a Gain rod cone log Gain 0 24 6-2 -4 -6 Sensitivity of Eye

8. Bahadir K. Gunturk8 Sensitivity of Eye

9. Bahadir K. Gunturk9 HDR The range of image capture devices is also low

10. Bahadir K. Gunturk10 HDR The range of image capture devices is also low

11. Bahadir K. Gunturk11 HDR HDR image rendered to be displayed on a LDR display.

12. Bahadir K. Gunturk12 HDR Problems: How to capture an HDR image with LDR cameras? How to display an HDR image on LDR displays?

13. Bahadir K. Gunturk13 Capture multiple images with varying exposure. Combine them to produce an HDR image.

14. Bahadir K. Gunturk14 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E (=total power per unit area)

15. Bahadir K. Gunturk15 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E z1 z2 E z1 = t1 * E z2 = t2 * E E1=z1/t1 E2=z2/t2 Estimates: Take a weighted sum of E1 and E2: w1 w2 E=( w1*E1 + w2*E2 ) / (w1+w2) E

16. Bahadir K. Gunturk16 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E z1 z2 E z1 = t1 * E z2 = t2 * E E1=z1/t1 E2=z2/t2 Estimates: Take a weighted sum of E1 and E2: w E=( w(z1)*E1 + w(z2)*E2 ) / (w(z1)+w(z2)) z 255

17. Bahadir K. Gunturk17 Creating HDR from Multiple Pictures Measured intensity, z t1 t2 t1 t2 Irradiance, E z1 z2 E z1 = t1 * E z2 = t2 * E E1=z1/t1 E2=z2/t2 Estimates: Take a weighted sum of E1 and E2: w E=( w(z1)*E1 + w(z2)*E2 ) / (w(z1)+w(z2)) z 255 Question: If t1 and t2 are not given, how can we estimate them?

18. Bahadir K. Gunturk18 Creating HDR from Multiple Pictures In general, the camera response is not linear. t1 t2 z1 = f ( t1 * E ) z2 = f ( t2 * E ) E1= g (z1) / t1 E2= g (z2) / t2 E=( w(z1)*E1 + w(z2)*E2 ) / (w(z1)+w(z2)) f g w z z Questions: How to estimate g and t?  One approach is based on polynomial model (Nayar). w is sometimes chosen as the derivative of f. (Mann)

19. Bahadir K. Gunturk19 Radiometric Self Calibration Polynomial model Exposure ratios: Cost function Solve using If exposure ratios are not known, solve iteratively Intensity Irradiance Pixel Image number

20. Bahadir K. Gunturk20 Tone Mapping Given an HDR image, how are we going to display it in an LDR display?

21. Bahadir K. Gunturk21 Tone Mapping Given an HDR image, how are we going to display it in an LDR display? Linear Nonlinear

22. Bahadir K. Gunturk22 Durand & Dorsey

23. Bahadir K. Gunturk23 Durand & Dorsey

24. Bahadir K. Gunturk24 Durand & Dorsey

25. Bahadir K. Gunturk25 Durand & Dorsey

26. Bahadir K. Gunturk26 Durand & Dorsey

27. Bahadir K. Gunturk27 Durand & Dorsey  Bilateral filter

28. Bahadir K. Gunturk28 Durand & Dorsey

29. Bahadir K. Gunturk29 Fattal et al in 1D 2500:1 log derivative 7.5:1exp integrate attenuate

30. Bahadir K. Gunturk30 Reinhard et al. L_white is the smallest luminance that will be mapped to pure white (1). Set L_white = L_max to have no “burn-out”.

31. Bahadir K. Gunturk31 Durand & Dorsey

32. Bahadir K. Gunturk32 Durand & Dorsey

33. Bahadir K. Gunturk33 Informal comparison Bilateral [Durand et al.] Photographic [Reinhard et al.] Gradient domain [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.] Gradient domain [Fattal et al.]

34. Bahadir K. Gunturk34 Spatially Varying Exposures Instead of capturing multiple pictures, allow different amounts of light pass for different pixel positions. Estimate the missing pixels. Combine to obtain an HDR image. 100%75% 50%25% Nayar

35. Bahadir K. Gunturk35 Image Reconstruction: Interpolation

36. Bahadir K. Gunturk36 Image Reconstruction: Aggregation

37. Bahadir K. Gunturk37 HDR image examples

38. Bahadir K. Gunturk38 HDR image examples

39. Bahadir K. Gunturk39 HDR image examples

40. Bahadir K. Gunturk40 The Bilateral Filter (BF) The SUSAN filter, which is essentially the bilateral filter, was used for corner/edge detection and denoising in [Smith & Brady 97]. The BF was presented in [Tomasi & Manduchi 98]. [Elad 02] and [Barash 02] show that the BF is related to the weighted least squares estimation and anisotropic diffusion. Fast implementations/approximations have been proposed, e.g., in [Paris & Durand 06]. In addition to image denoising, the BF is used in tone mapping of HDR images, contrast enhancement, 3D mesh smoothing, blocking artifact reduction, etc.

41. Bahadir K. Gunturk41 Bilateral Filtering Intensity (range) proximity Spatial (domain) proximity

42. Bahadir K. Gunturk42 Bilateral Filtering Input Gaussian Bilateral

43. Bahadir K. Gunturk43 What are the optimal values of the parameters of the Bilateral Filter? MSE=49.8MSE=50.9 MSE=30.3MSE=43.4 MSE=42.5MSE=71.5 MSE=100.0