Color transfer between high-dynamic-range images H. Hristova, R. Cozot, O. Le Meur, K. Bouatouch University of Rennes 1 Rennes, France

Outline Introduction Main objective Contributions Extension to the HDR domain of a color transfer method Results and evaluation Generalization for state-of-the-art color transfer methods Conclusion Outline of the presentation: Introduction to the main objective and the main contributions of the paper Presenting an extension to the HDR domain of a novel local color transfer method (Followed by) results and evaluation Generalizing the developed extension to other state-of-the-art color transfer methods + results 2

Main goal Carrying out a color transfer between two HDR images directly in the HDR domain Our main goal is to carry out a color transfer between two HDR images directly in the HDR domain. -Why? -Because tone mapping operators compress the information contained in HDR images and therefore the plausibility of the tone-mapped images cannot be guaranteed. Solution to the problem: to apply color transfer methods between two HDR images. Color transfer methods build a transformation mapping between two LDR images and aim at “borrowing” the look (mood) of a reference image. Can color transfer methods be directly applied to the HDR domain? -> next slide Input Reference Solution: apply color transfer methods to stylize an HDR image with regards to a reference image 3

Why do LDR color transfer methods need to be extended to the HDR domain? LDR color spaces - well predict the color gamut for luminance levels between zero and the display white point - uncertain applicability to HDR images Color trend above the perfect diffuse white The color transfer methods need to be extended to HDR images. -Why? -First: Traditional LDR color spaces, such as lab and CIE Lab, which are widely used in color transfer algorithms, predict well the color gamut for luminance levels below the display white point. However, when it comes to luminance levels way beyond the display white point, like those of HDR images, the applicability of these color spaces becomes uncertain. For example, when the lightness reaches high levels (above 255 in the figure), the imagery primaries (x, y) ranging from 0 to 1 cannot depict and predict the color gamut for these high levels of luminance. To predict the color trend above the perfect diffuse white, we need to perform some modifications to traditional color spaces. 4

Why do LDR color transfer methods need to be extended to the HDR domain? Assumption: a unique multivariate Gaussian distribution HDR domain: to fit the high range of lightness of HDR images we need to assume mixture of Gaussian distributions Second: most color transfer methods build their transformations based on the assumption that a multivariate Gaussian distribution is sufficient to fit the luminance and chroma variations. However, this is hardly true in the LDR domain. The latter holds for the HDR domain. The high luminance range of an HDR image cannot be fitted only by one Gaussian distribution. Therefore, we need to assume a mixture of Gaussian distributions. 5

Why do LDR color transfer methods need to be extended to the HDR domain? Lightness - approximated by luminance in the LDR domain HDR domain - distinguish between the absolute luminance and the lightness (the L channel of CIE Lab) Finally: Usually, the luminance is approximated by the lightness (the L channel from CIE Lab color space) in the LDR domain. Nevertheless, there is a distinguish between the luminance and the lightness in the HDR domain. On one hand, the luminance of HDR images is presented by the Y channel in XYZ color space and it is called absolute luminance. On the other hand, lightness is represented by the L channel of CIE Lab color space. Both of them are not equal in the context of HDR imagery. Therefore, color transfer methods which use these two notions in their algorithms need to be properly adapted to the HDR domain. 6

Contributions Adaptation of [Hristova et al., 2015] color transfer method to HDR images - HDR color spaces - Modifications of the clustering step and of the image classification Cluster-based local chromatic adaptation transform Generalization for state-of-the-art color transfer methods Our main contributions are: the development of extension to HDR images of a new local color transfer method [Hristova et al., 2015], consisting of: - modifications of traditional color spaces (their replacement with HDR-based ones) - modifications of the clustering and classification algorithms of the local method: distinguishment between lightness and luminance The development of novel cluster-based local chromatic adaptation transform for adapting the colors of the input HDr image to a reference white point. 7

Extension to HDR images Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] Linear search for significant peaks in the image hue histogram The number of significant peaks determines the cluster number - Colors-based style images: hue histogram - Light-based style images: luminance histogram The local method by [Hristova et al., 2015] is a new method for color and style transfer consisting of several steps. These steps (either independently or as a combination) can be used in other color transfer methods. Therefore, we will first present the extension of [Hristova et al., 2015] and then, we will generalize that extension to other color transfer methods. [Hristova et al., 2015] starts with a conversion to CIE Lab color space. Once both images are converted to that color space, they are classified according to their main features (either colors or light) as shown on the figure. Two types of images are considered: colors-based and light-based ones. The classification boils down to finding significant peaks in the hue histogram of both images. During the classification step, we determine also the number of clusters which will be passed to the clustering process. On one hand, the number of color clusters is determined by the hue histogram. On the other hand, the number of luminance clusters is defined by the number of significant peaks in the luminance histogram of the image. After the classification, both images are clustered to Gaussian clusters and then, the input/reference clusters are mapped. Four policies are developed to map the clusters. They take into account both the luminance histogram and the luminance-hue distributions of images. A color transfer is carried out between each pair of corresponding clusters. An optimal transformation is used (Monge-Kantorovich optimization problem). Finally, a chromatic adaptation transform is applied to adapt the colors of the output image to the reference illuminant. 8

Extension to HDR images LDR images CIE Lab L channel of CIE Lab Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] HDR images hdr-CIELab Log- luminance Dashed line: cubic function of L channel (CIE Lab) Solid line: Michaelis-Menten function by which we replace the cubic function of L channel (CIE Lab) hdr-CIELab color space [Fairchild et al., 2004] The following extension of [Hristova et al., 2015] is proposed. The extension is applied to HDR images instead to LDR images. One of the issues of the direct applying of the method to HDR images has been already discussed: the uncertainty in predicting the color gamut for high luminance values. Therefore, instead of using the LDR CIE Lab color space, we perform modifications to its channels to derive hdr-CIELab color space [Fairchild et al., 2004]. The cubic function of the L channel (the dashed line) is replaced by Michaelis-Menten function (the solid line) for better approximation of the lightness in HDR images. Moreover, this modification influences the predictability of the color trend for HDR images. Furthermore, as the image classification depends strongly on the number of significant peaks in the image hue histogram, if we ensure a better predictability of the image hue, then we ensure the successful classification of images and the successful estimation of the number of color clusters (as the number of significant peaks corresponds to the number of color clusters). How about the number of light/luminance clusters? -> we adopt log-luminance histogram in the place of the L channel histogram -> explanation on the next slide. [Fairchild et al., 2004] 9

Extension to HDR images LDR images CIE Lab L channel of CIE Lab L-based clustering Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] HDR images hdr-CIELab Log- luminance Log- luminance clustering To find the optimal number of clusters for an image during the classification step, we need to find number of significant peaks in the luminance histogram of the image. Moreover, light-based images are clustered according to the luminance histogram. Finally, three of the mapping policies are based on luminance histogram information. In the LDR domain we can use the L channel of CIE Lab color space in the place of the luminance. However, in the HDR domain we use the absolute luminance and more precisely, the logarithmic transform of the absolute luminance (which we call log-luminance). The log-luminance is a good approximation of the brightness. Furthermore, the logarithmic transform preserves the regions of minima and maxima of a histogram and therefore we are able to recover the shadows, midtones and highlights from the log-luminance histogram. To this end, log-luminance histogram is used in the classification and clustering steps of the algorithm in the place of the lightness channel of hdr-CIELab color space. Once we carry out the image clustering and we map the reference to the input clusters, we apply an optimal transformation between each pair of corresponding clusters. We call the obtained image an output. The last step adapts the colors of the output image to the reference illuminant. Logarithmic transform 10

Extension to HDR images LDR images CIE Lab L channel of CIE Lab L-based clustering Local CAT Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] HDR images hdr-CIELab Log- luminance Log- luminance clustering Cluster- based local CAT In the LDR domain, CAT algorithm adapts the input image to a global reference point. The novel cluster-based local CAT algorithm, however, first clusters both images into regions (highlights, midtones, shadows). We search for significant peaks in the log-luminance histogram and then, we define the region limits as the minima between two significant peaks. 11

Extension to HDR images LDR images CIE Lab L channel of CIE Lab L-based clustering Local CAT Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] HDR images hdr-CIELab Log- luminance Log- luminance clustering Cluster- based local CAT (h) (sh) (m) (h) Like in the LDR domain, we apply a low-pass Gaussian filter to the input image to compute locally (like a “white” image) the input illuminant. Then, we cluster the input image into regions. Each region corresponds to part of the “white” image which we take into account to carry out the transform region-wisely. As far as the reference image is concerned, we estimate a representative white point for each reference region. Then, this white point becomes a global white point for the region. The reference white points for each region are shown in blue dashed lines. (m) Gaussian low-pass filter (sh) 12

Extension to HDR images LDR images CIE Lab L channel of CIE Lab L-based clustering Local CAT Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] HDR images hdr-CIELab Log- luminance Log- luminance clustering Cluster- based local CAT Here is a video example of how the result changes over the iterations. Around the 10th iteration, the input image is overall adapted to the reference illuminant. However, there is a slight cast of the greenish input reference on the result. It is removed at the end of the iteration. Input Cluster-based local CAT Reference 13

Extension to HDR images LDR images CIE Lab L channel of CIE Lab L-based clustering Local CAT Input and reference images Color space conversion Image classification Clustering and mapping Color transfer Chromatic adaptation transform Final result [Hristova et al., 2015] HDR images hdr-CIELab Log- luminance Log- luminance clustering Cluster- based local CAT Another example of illuminant adaptation (video). Input Cluster-based local CAT Reference 14

Objective evaluation of the results 10 image pairs Two tone-mapping operators: [Durand et al., 2002] and [Reinhard et al., 2002] SSIM and Bhattacharya coefficient We used 10 image pairs to obtain 10 HDR image results for the original [Hristova et al., 2015] method and its extension. We evaluated the results in the LDR domain after tone-mapping them. SSIM and Bhattacharya coefficient are two complementary metrics (used also by [Hristova et al., 2015]) measuring respectively the degree of artifacts and how successful the color transfer is. The HDR extension obtains the highest scores for both metrics and for both tone-mapping operators. 15

Results [Hristova et al., 2015] HDR extension Input Color transfer with CAT Color transfer with CAT The results support the objective evaluation. The first column represents the input and reference images. The second column shows the results obtained with the method [Hristova et al., 2015] directly applied to HDR images (with and without local CAT). The third column shows the results obtained with the HDR extension of the method (with and without local CAT). If we apply [Hristova et al., 2015] directly to HDR images, then the method it opt for artifacts. Moreover, the details are lost. On the other hand, the HDR extension manages to preserve the details and properly transfers the reference colors to the output image. Reference Color transfer without CAT Color transfer without CAT 16

Generalization and results Input Reference Now we show several results for state-of-the-art methods. The first row consists of the input and reference images. The first image on the second row is the result of the direct applying of the state-of-the-art method to HDR images and the second image on the second row presents the result of the same state-of-the-art method, modified in order to enhance the color transfer between two HDR images. [Reinhard et al., 2002] is a global method which assumes that the light and colors of the image can be fitted by a unique Gaussian distribution. As discussed, one Gaussian distribution is not sufficient to fit the high luminance range of HDR images. Therefore, to enhance the effect of the color transfer, we recommend to carry out Reinhard’s transformation locally on pairs of clusters. To this end, we recommend [Tai et al., 2005] to be used in the place of [Reinhard et al., 2005] in the HDR domain. [Tai et al., 2005] - clustering (local transformations) [Reinhard et al., 2001] - global method 17

[Pitié et al., 2007] - hdr-CIELab Generalization and results Input Reference [Pitié et al., 2007] is a global method which applies optimal mapping between the input and reference images. In the LDR domain, the methods adopts CIE Lab color space. As we can see from the first result on the figure, CIE Lab is less effective from this type of transformation between HDR images than hdr-CIELab. The latter color space predicts better the color palette of the reference image and manages to transfer it better to the final result. [Pitié et al., 2007] - CIE Lab [Pitié et al., 2007] - hdr-CIELab 18

Generalization and results Input Reference As a local method, [Bonneel et al., 2013] performs clustering to both input and reference images. The clustering is luminance-based. To extend the method to HDR images, we replace the luminance-based clustering with log-luminance-based clustering (as shown for [Hristova et al., 2015] earlier in the presentation). Moreover, instead of carrying out the transformation in CIE Lab color space, we adopt hdr-CIELab color space in the HDR domain. That way, the proposed extension lessens the visual artifacts in the final result (in comparison to the direct application of [Bonneel et al., 2013]). [Bonneel et al., 2013] - log-luminance clustering [Bonneel et al., 2013] - luminance clustering 19

Conclusion Extension of a novel local color transfer method [Hristova et al., 2015] Modifications to CIE Lab -> hdr-CIELab Luminance/Lightness -> Log-luminance Generalization to state-of-the-art methods Future work Need for a more precise color mapping/color transformation between two HDR images HDR color spaces 20

Thank you for your attention! 21