1. Physics-based Illuminant Color Estimation as an Image Semantics Clue Christian Riess Elli Angelopoulou Pattern Recognition Lab (Computer Science 5) University of Erlangen-Nuremberg November 8, 2009

2. Page 2 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Illumination as an Image Semantics Clue Semantic analysis: Cues on objects, location, time Can we add illumination color as a clue?

3. Page 3 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Existing Methods in Illuminant Estimation Machine Learning-based, e.g.  Gamut Mapping (e.g. Forsyth)  Recent Gray Edge Variants (e.g. Gijsenij, Gevers)  Color by Correlation (e.g. Finlayson) Physics-based, e.g.  Dichromatic Reflectance-based (e.g. Klinker et al.)  Intersection of diffuse color planes (e.g. Finlayson)  Inverse-Intensity Chromaticity (Tan et al.) Require proper training Require “clean” input [1] Tan, Nishino, Ikeuchi. “Color Constancy through Inverse-Intensity Chromaticity Space,” Journal of the Optical Society of America A, 21(3): 321-334, 2004. [1]

4. Page 4 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Goal Show that physics-based methods can be used  On arbitrary images,  Maybe sacrificing the exact estimate,  But giving higher level information (semantics).

5. Page 5 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue The Inverse Intensity-Chromaticity Approach Neutral Interface Assumption (NIA): specular color = illuminant Image formation Let  Chromaticity  Diffuse chromaticity  Specular chromaticity  Rewritten w/ chromaticities Diffuse geometry Diffuse reflectance Specular geometry Specular reflectance

6. Page 6 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue The Inverse Intensity-Chromaticity Approach Tan et al. showed, where NIA: specular color = light color =. The illuminant chromaticity relates linearly to the sum of intensities and the pixel chromaticities. Specular regions form a triangle in inverse-intensity space, pointing to the illuminant color.

7. Page 7 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Towards Automated Scene Assessment Tan et al. propose to estimate the illuminant chromaticity using the Hough transform along the y-axis. Hypothesis: The histogram shape gives clues on the illuminant estimate quality, and the illumination environment in general. Hough Space along y-axis Hough Space, as a histogram in its own right

8. Page 8 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Influence of the Specularity Segmentation Experiments with different specularity segmentations R, G, B estimates surprisingly stable Noise component shows as a broader peak.

9. Page 9 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Experiments with similar scenes Illuminant color estimates not sufficiently reliable: Left image illuminant color estimate (0.378, 0.303, 0.318), Right image illuminant color estimate (0.339, 0.341, 0.319). Influence of the Specularity Segmentation

10. Page 10 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Towards automated Self-assessment Is it possible to detect unmet constraints (e.g. poor specularity segmentation)? Shape of the histogram peak  Fit Gaussian to peak  Fit triangle to peak  Optimization criteria  Intersection area  Sum of squared differences

11. Page 11 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Multiple illuminants mix in the histogram  Observed peak might be a mixture of noise components from multiple light sources.  Decomposition into ROIs shows the mixing property. Further Challenges on the Histogram Shape

12. Page 12 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Barely specularities, high amount of noise  Indoor illumination, reddish interreflections  Red/blue channels: Peak can be “hidden” Further Challenges on the Histogram Shape

13. Page 13 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Summary This is preliminary work. Shape of the histogram can be used as a semantic source of information. Method is insensitive to variations in specularity segmentation. Further analysis of histogram shape:  Impact of binning  Noise vs. multiple illuminants  Learn histogram shapes?  Iterative simultaneous segmentation and illuminant estimation

14. Page 14 Christian Riess Nov. 8, 2009Physics-based Illuminant Color Estimation as an Image Semantics Clue Image Sources http://www.flickr.com http://www.walks.ru Barnard et al.: A Comparison of Computational Color-Constancy Algorithms, IEEE TIP 11(9), 985- 996, 2002. Thank you for your attention!