1. What is the Space of Camera Responses? Michael Grossberg and Shree Nayar CAVE Lab, Columbia University IEEE CVPR Conference June 2003, Madison, USA Partially funded by NSF ITR Award, DARPA/ONR MURI

2. The Camera Response Scene Radiance Linear Function (Optical Attenuation) Image Irradiance Ls E 0255 Non-Linear Camera Response Image Intensity B f

3. Impact of Camera Response Accurate scene radiance required for Color Constancy Creating Accurate High Dynamic Range Images Photometric Stereo Shape from Shading Inverse Rendering Measuring BRDF from Images

4. Response Model for Recovery Charts: Known Reflectance Known Reflectance [Sawchuk, 77 Chang and Reid, 96] [Debevec and Malik, 1997, Mann, 2000, Mann, 2000, Mann and Picard, 1995, Mitsunaga and Nayar, 1999, Tsin et al., 2001] Multiple Images: Changing Exposure Changing Exposure Breaking Ambiguities [ Grossberg, Nayar, 2002] Model Required for Interpolation

5. Response Normalization and Monotonicity Irradiance E Intensity B Dark Current Level Saturation Level Normalize Monotonicity Key Property: Makes response invertible Irradiance E 0 0 1 1 f Camera Response

6. Space of Response Functions Let The space of all functions with Inequalities: Cone of monotonic functions 01 Space of theoretical response functions

7. Linear Model of Response f0f0 h f M-order linear approximation model: M-order linear approximation model: base responseparameters of modelbasis functions h1h1 h2h2 f0f0 f

8. Choosing a Basis Possible basis h 1, h 2, … Possible basis h 1, h 2, … Which basis is best? Which basis is best? – Depends on which response functions occur Irradiance Intensity h1h1h1h1 h2h2h2h2 h3h3h3h3 h4h4h4h4 Polynomial basis, h4h4h4h4 Irradiance Intensity h2h2h2h2 h1h1h1h1 h3h3h3h3 Trigonometric basis,

9. Film Positive, negative, consumer, professional, color, b/w Agfacolor Futura Agfachrome RX-II Fuji F125 Fuji FDIC Kodak Advanced Kodak Gold … Database of Response Functions (DoRF) Collected 201 response curves from : CCDs Kodak's KAI and KAF series … Digital/Video Sony DC 950 Canon Optura Gamma curves …

10. Sample curves Normalized Brightness Agfachrome CTPrecisa100 Green Agfachrome RSX2 050 Blue Agfacolor Futura 100 Green Agfacolor HDC 100 plus Green Agfacolor Ultra 050 plus Green Agfapan APX 025 Agfa Scala 200x Cannon Optura Fuji F125 Green Fuji F400 Green Kodak Ektachrome-100plus Green Kodak Ektachrome-64 Green Kodak KAF2001 CCD Kodak KAI0372 CCD Kodak Max Zoom 800 Green Sony DXC-950 Irradiance gamma curve, g =0.6 gamma curve, g =1.0 gamma curve, g =1.4 gamma curve, g =1.8 Kodak DCS 315 Green Intensity 00.10.20.30.40.50.60.70.80.91 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Evaluate Bases Using DoRF: Good basis provides good approximation with few parameters

11. Empirical Model of Response (EMoR) Build Basis using DoRF 175 training curves, 26 testing curves Apply PCA to DoRF f 0, h 1, h 2, …. 99.5% of energy in first 3 dimensions Normalized Response Normalized Response 00.20.40.60.81 -0.04 0 0.04 -0.08 Irradiance Intensity 00.20.40.60.81 0 0.2 0.4 0.6 0.8 1 Mean CurvePrincipal Components Percent of Energy Percent h1h1 h2h2 h3h3 h4h4 Principal Components 23456789101 Energy Irradiance Intensity 80 82 84 86 88 90 92 94 96 98 100

12. h 1 h 2 h 3 h 4 00.20.40.60.81 0 0.2 0.4 0.6 0.8 1 Mean Curve Irradiance Intensity 00.20.40.60.81 Irradiance Intensity Principal Components -0.04 0 0.04 -0.08 Percent 80 82 84 86 88 90 92 94 96 98 100 Principal Components 23456789101 Energy Log basis: Log model Generalizes gamma curves Log EMoR Apply PCA to Log DoRF 99.6% of energy in first 3 dimensions Log Model of Response (Log EMoR) base response parameters of model basis functions

13. Monotonic Approximation Monotonicity:  Linear inequalities in c n  Linear inequalities in c n Derivative at Unity - 5051015202530354045 Gamma Curves Other DoRF Curves Gamma = 0.2 Gamma = 2.5 6 5 4 3 2 1 0 Derivative at Origin Derivative at Unity Monotonic functions Least Squares error: Quadratic Programming 2 Principal Components

14. EMoR/Log EMoR Model Evaluation 4.12E-02 1.94E-02 8.87E-03 4.15E-03 2.82E-03 1.91E-03 1.58E-03 1.15E-03 9.10E-04 7.60E-04 6.02E-04 9.07E-02 4.77E-02 2.60E-02 1.51E-02 1.05E-02 8.09E-03 7.46E-03 4.96E-03 4.30E-03 3.95E-03 3.12E-03 1 2 3 4 5 6 7 8 9 10 11 Mean RMSE Mean Disparity Parameters EMoR Model Accuracy: 6.8 bits 9.0 bits8.3 bits Mean Disparity 7.04E-02 1.71E-02 8.83E-03 4.99E-03 3.44E-03 2.80E-03 2.52E-03 1.67E-03 1.36E-03 1.79E-03 9.54E-04 Log EMoR Model 1 2 3 4 5 6 7 8 9 10 11 Mean RMSE Parameters 1.11E-01 3.44E-02 1.85E-02 1.11E-02 8.14E-03 6.58E-03 5.77E-03 4.12E-03 3.36E-03 4.14E-03 2.49E-03 Accuracy: 6.8 bits 8.4 bits8.6 bits

15. Models Compared 1 2 3 4 5 6 7 3.46E-02 N. A. N. A. N. A. N. A. N. A. N. A. 7.37E-02 3.29E-02 1.71E-02 1.06E-02 6.93E-03 4.95E-03 3.65E-03 6.83E-02 3.91E-02 2.58E-02 1.89E-02 1.44E-02 1.16E-02 9.46E-03 4.00E-02 1.73E-02 6.27E-03 2.54E-03 1.77E-03 1.07E-03 9.55E-04 Model Gamma Polynomial Trigonometric EMoR Parameters Accuracy: 4.8 bits5.9 bits5.3 bits7.3 bits RMSE Error 1 2 3 4 5 6 7 2.52E-01 N. A. N. A. N. A. N. A. N. A. N. A. 4.22E-01 2.90E-01 2.12E-01 1.51E-01 1.16E-01 8.96E-02 7.55E-02 4.12E-01 3.30E-01 2.74E-01 2.32E-01 2.01E-01 1.77E-01 1.57E-01 3.04E-01 1.38E-01 8.54E-02 4.90E-02 3.15E-02 2.25E-02 1.86E-02 Model Gamma Polynomial Trigonometric EMoR Parameters Accuracy: 2.0 bits2.2 bits1.9 bits3.9 bits Disparity Error

16. Normalized Brightness Other chart values Chart values used for fit Monotonic EMoR Monotonic polynomial EMoR Polynomial Response from Sparse Samples 00.10.20.30.40.50.60.7 1 0.90.8 Normalized Irradiance 0.2 0.4 0.6 0.8 1 0 Camera Response Normalized Intensity

17. Response from Multiple Images Inverse Camera Response Data from chart Monotonic EMoR Mitsunaga-Nayar (Polynomial) Debevec-Malik = 8 (Log space) Debevec-Malik = 32 (Log space) Debevec-Malik = 128 (Log space) 00.10.20.30.40.50.60.70.80.91 0 0.2 0.4 0.6 0.8 1 Normalized Irradiance Normalized Intensity

18. Summary Determined Space of Response functionsDetermined Space of Response functions –Intersection of cone and plane Linear and Log approximation modelsLinear and Log approximation models –Generalized previous models Database of Response Functions (DoRF)Database of Response Functions (DoRF) –Evaluate models Empirical Model of Response (EMoR)Empirical Model of Response (EMoR) –Superior model of camera response based on DoRF DoRF and EMoR available for downloadDoRF and EMoR available for download from www.cs.columbia.edu/CAVE from www.cs.columbia.edu/CAVE

19. Colors