Keywords

1 Introduction

As a fundamental image processing technique, color transfer can impose the color characteristics of reference image on source image. Recently, a lot of global or local approaches have been proposed to achieve the transfer of color patterns. However, most of them often produce low-fidelity transfer results due to the destruction of color, detail and illumination. As shown in Fig. 1(c)–(f), Reinhard et al.’s approach [1] can compute and transfer the color mean and standard deviation of two images, but this statistics-based approach cannot always yield a high-quality transfer result; Pitie et al.’s approach [2] can execute histogram matching by a n-dimensional probability density function, but it also fails to improve the transfer quality due to the low-fidelity artifacts; Pouli et al.’s approach [3] can reshape histograms at different scales for a better transfer control, but it doesn’t reverse the detail and illumination problems; Wu et al.’s approach [4] can achieve a semantic color transfer using high-level scene content analysis, but this content-based approach doesn’t still consider how to eliminate the low-fidelity artifacts. Therefore, we believe that an ideal color transfer approach should provide the high fidelity in terms of color, detail and illumination. Obviously, its effectiveness depends on how to preserve the scene detail and harmonize the illumination environment in color transfer.

Fig. 1.
figure 1

Comparison of color transfer approaches. (a) Source image. (b) Reference image. (c) The result by Reinhard et al.’s approach. (d) The result by Pitie et al.’s approach. (e) The result by Pouli et al.’s approach. (f) The result by Wu et al.’s approach. (g) The result by our approach with detail preserving. (h) The result by our approach with detail preserving and illumination awareness. Note: The traditional approaches cannot yield the high-fidelity color transfer results in terms of detail and illumination, e.g., The scene detail and illumination environment are severely destroyed in (c) and (d); although Pouli et al.’s approach and Wu et al.’s approach can reshape the histograms and achieve the content-based analysis effectively, its color, detail and illumination isn’t still satisfying in (e) and (f), especially in the part of woods and house. In contrast, according to the scene detail and the illumination environment in (a), our unified fidelity optimization model can generate a high-fidelity transfer result with detail preserving and illumination awareness in (g) and (h).

Full size image

On the whole, corresponding to the color, detail and illumination characteristics, we define our novel color transfer as a unified optimization problem with three energy terms: color mapping, detail preserving and illumination awareness. Color mapping can impose the color style of one image on another image using histogram matching; Detail preserving can maintain the scene detail in an image by gradient guidance; Illumination awareness can harmonize the whole illumination environment based on illumination affinity. As shown in Fig. 1(g) and 1(h), compared with the other four approaches, our illumination-aware color transfer can generate a high-fidelity transfer result with the unification of color, detail and illumination by preserving the scene detail and harmonizing the illumination environment. Moreover, based on the above three characteristics, we also propose an objective evaluation metric to analyze the example-based color transfer approaches.

In summary, our specific contributions in this paper are:

  • A novel color transfer approach which can achieve a high-fidelity transfer by the combination of color mapping, detail preserving and illumination awareness.

  • A simple and effective objective evaluation metric which can measure the fidelity of color transfer in terms of color, detail and illumination.

After a brief review of related work in next section, we elaborate our fidelity optimization approach in Sect. 3. Our new evaluation metric is introduced in Sect. 4. The related experiment results are shown in Sect. 5. A final conclusion and discussion about this paper are given in Sect. 6.

2 Related Work

During the past few years, a lot of color transfer approaches have been proposed to achieve the example-based editing in color style. Reinhard et al. [1] first proposed a global approach to transfer one image’s color style to another. Its effectiveness depends on the composition similarity between source and reference images. Chang et al. [5, 6] proposed a color transfer approach based on the eleven color categories, and later extended it to process video data. Pitie et al. [2] proposed a $n-$ dimensional pdf approach for color transfer, which can guarantee the preservation of color distribution. Pouli et al. [3] proposed a novel histogram reshaping technique which can transfer the color palette between images of arbitrary dynamic range. Wu et al. [4] proposed a content-based method for transferring the color patterns between images, which can put an emphasis on high-level scene content analysis instead of color statistics in the previous methods.

Tai et al. [7, 8] first proposed a local color transfer approach using a soft color segmentation algorithm. Luan et al. [9] developed an interactive tool for local color transfer which is based on the observation that color transfer operations are local in nature while at the same time should adhere to global consistency. Wen et al. [10] developed a stroke-based color transfer system, which can specify some local regions to transfer their color styles by a stroke-based user interface. An et al. [11] proposed a user-controllable approach for color transfer, which can construct the transfer constraints by specify the corresponding regions in the source and reference images.

In conclusion, these approaches can achieve a variety of color transfer using the global or local color distributions. However, most of them cannot simultaneously provide the fidelity in terms of color, detail and illumination characteristics, especially a lack of detail preserving and illumination awareness for color transfer. Thus our optimization mainly expects to preserve the scene detail and harmonize the global illumination environment for the high-fidelity transfer.

3 Our Optimization Model

For a high-fidelity result, color, detail and illumination should be fully guaranteed in the process of transfer. Thus, corresponding to these three characteristics, our color transfer algorithm will focus on color mapping, detail preserving and illumination awareness. Color mapping can employ histogram matching to impose color style of reference image on source image; Detail preserving can apply gradient guidance to maintain scene detail in source image; Illumination awareness can introduce intrinsic images to harmonize illumination environment.

Formally, we describe the above color transfer as an optimization problem, which constructs and minimizes an energy function with three different terms:

$$ \Phi = {\text{argmin}}\{ w_{c} E_{c} ({\text{H}}\left( \Phi \right),{\text{H}}({\text{R}})) + w_{d} E_{d} ({\text{G}}(\Phi ), {\text{G}}({\text{I}})) + w_{i} E_{i} ({\text{S}}(\Phi ), {\text{S}}({\text{I}})) \} $$
(1)

where E c , E d and E i represent three energy terms of color, detail and illumination; Φ, I and R indicate output result, source image and reference image respectively; \( {\text{H}}( \cdot ) \) denotes the color histogram of one image; \( {\text{G}}( \cdot ) \) denotes the gradient map of one image; \( {\text{S}}( \cdot ) \) denotes the illumination component of one image; w c , w d and w i are three coefficients for weighting the importance of color, detail and illumination, usually fix w c , w d , w i to 1. In the next sections, we will introduce these three energy terms in detail.

3.1 Color Mapping

In Eq. 1, we define the energy term E c for color mapping, which is expected to generate an initialized transfer result. Generally, using color histograms, we formulate E c as the following cost function:

$$ E_{c} ({\text{H}}(\Phi ), {\text{H}}({\text{R}})) = \mathop \sum \limits_{k}^{n} ({\text{H}}_{k} \left( \Phi \right) - {\text{H}}_{k} ({\text{R}}))^{2} $$
(2)

where k is the histogram bin; n is the number of histogram bins. For a sufficient color mapping, Eq. 2 requires the output result to match its reference image in term of color histogram as much as possible.

Obviously, since the function \( {\text{H}}( \cdot ) \) is a statistical operation on the entire image, Eq. 1 can hardly integrate E c as a part of linear system in the process of solving. Therefore, we first calculate an intermediate image using histogram matching between source image and reference image. Then, in order to avoid the cross-impact of color and detail, we weaken its detail by a WLS filter [12], which can efficiently split color and detail from an image. Finally, we rewrite Eq. 2 as \( \mathop \sum \limits_{p} (\Phi_{p} - F_{p} )^{2} \), where F is an initialized transfer result with color mapping.

3.2 Detail Preserving

Unfortunately, color mapping cannot directly produce a high-fidelity transfer result due to the loss of scene detail. Thus Eq. 1 declares the energy term E d to preserve the detail characteristic on source image. Here, we define E d as the following gradient-guided function [13]:

$$ E_{d} ({\text{G}}(\varPhi ), {\text{G}}({\text{I}})) = \mathop \sum \limits_{p}^{m} \nabla \varPhi_{p} - \lambda \nabla {\text{I}}_{p}^{2} $$
(3)

where \( \nabla \) is a gradient operator; p is an image pixel; m is the number of image pixels; λ is a coefficient for adjusting image contrast, i.e., λ>1 can increase contrast; λ<1 can decrease contrast; λ=1 can maintain contrast. Specifically, Eq. 3 serves to bind the output result and the source image in the gradient domain.

Generally, the gradient-domain optimization can effectively maintain the original features of source image by manipulating the pixel differences. Thus, for detail preserving, we impose the gradient of output result to match the gradient of source image. First of all, we employ Sobel difference operator to compute the gradient of source image. Then, according to user’s sharpness demand, we further determine the value of contrast coefficient. Finally, as a constraint in scene detail, Eq. 3 can be integrated into Eq. 1 in order to achieve a high-quality detail preserving.

3.3 Illumination Awareness

Besides detail preserving, a harmonious illumination environment can also improve the fidelity of transfer result. Therefore, we expect to perceive the whole illumination environment of source image and apply it to the final transfer result. Here, we introduce an illumination affinity [14] based on intrinsic images to achieve illumination awareness. As a typical image decomposition technique, intrinsic images can decompose an image into the product of an illumination component that represents lighting effects and a reflectance component that is the color of the observed material, which has been widely applied to many computational photography applications [14–18], such as relighting, recoloring, retexturing and so on. According to a planar reflectance assumption on local windows, the illumination affinity L can be defined as a large sparse matrix:

$$ {\text{L}}\left( {{\text{i}},{\text{j}}} \right) = \mathop \sum \limits_{{k|(i,j) \in W_{k} }} N_{k}^{T} N_{k} (i_{k} ,j_{k} ) $$
(4)

where L(i, j) is the (i, j)th element of L; i k and j k are the local indices of i and j in the local window W k ; N k is the matrix \( I_{d} - M_{k} (M_{k}^{T} M_{k} )^{ - 1} M_{k}^{T} \) with the identity matrix I d , N T k $ is the transpose matrix of N k ; M k is a (|W k |+3)\( \times \) 3 matrix, M T k is the transpose matrix of M k ; for each k \( \in \) W k , M k contains a row of the form [I r k , I g k , I b k ], and the last three rows of M k are of the forms [\( \sqrt \varepsilon \), 0, 0], [0, \( \sqrt \varepsilon \), 0], and $[0, 0\( \sqrt \varepsilon \)]; [I r , I g , I b] are the RGB components of the source image I, \( \varepsilon \) is a regularizing parameter, and |W k | is the number of pixels in the window W k .

The affinity in Eq. 4 can accurately describe the actual illumination environment on source image, which will provide a perfect illumination reference for the high-fidelity color transfer. Thus we can employ it to guarantee the illumination harmonization in the process of transfer. Here, based on L, the energy term E i in Eq. 1 can be defined as the following cost function:

$$ E_{i} \left( {{\text{S}}\left( \varPhi \right), {\text{S}}\left( {\text{I}} \right)} \right) = \mathop \sum \limits_{i}^{m} \mathop \sum \limits_{j}^{m} L\left( {i,j} \right)\left( {s_{i} - s_{j} } \right)^{2} = \mathop \sum \limits_{i}^{m} \mathop \sum \limits_{j}^{m} L(i,j)\left( {\frac{{\varPhi_{i} }}{{r_{i} }} - \frac{{\varPhi_{j} }}{{r_{j} }}} \right)^{2} $$
(5)

where s and r are the illumination and reflectance components of Φ respectively. Equation 5 demonstrates that it is very difficult to directly solve Φ in the condition of s. Therefore, according to intrinsic images, we exchange s with Φ so that Eq. 1 can be represented as a uniform solution about Φ. To simplify the reflectance estimation, we further assume that the reflectance r of the final result Φ is the same with the reflectance r* of the initialized result F, i.e., r = r*. At last, we use its global sparsity prior to efficiently generate r*. After obtaining L and r, Eq. 5 can also be integrated into Eq. 1 for illumination awareness. Moreover, to analyze the effect of illumination coefficient w i in Eq. 1, we compare some transfer results with different coefficient in Fig. 2, which demonstrates that the adjustment of coefficient can effectively enhance the harmonization in term of illumination characteristics.

Fig. 2.
figure 2

Comparison of illumination coefficients. (a) The result with w i = 0. (b) The result with w i = 0.01. (c) The result with w i = 0.1. (d) The result with w i = 1. Note: Along with the increase of coefficient from (a) to (d), The harmonization of illumination is effectively improved in the left part of woods, which can yield a color transfer result with illumination fidelity.

Full size image

4 Evaluation Metric

Recently, some objective metrics [13, 19–21] have been proposed to evaluate the performance of color transfer approaches. However, most of them only consider measuring the differences of color or structure properties. In contrast, our metric can evaluate their performance in terms of color, detail and illumination, which are three key characteristics for a high-fidelity transfer result. Formally, we define the metric as the following MSE function:

$$ \frac{{w_{c} }}{n}\mathop \sum \limits_{k}^{n} ({\text{H}}_{k} \left( \varPhi \right) - {\text{H}}_{k} ({\text{R}}))^{2} + \frac{{w_{d} }}{m}\mathop \sum \limits_{p}^{m} ({\text{G}}_{p} \left( \varPhi \right) - {\text{G}}_{p} ({\text{I}}))^{2} + \frac{{w_{i} }}{m}\mathop \sum \limits_{p}^{m} ({\text{S}}_{p} \left( \varPhi \right) - {\text{S}}_{p} ({\text{I}}))^{2} $$
(6)

where \( {\text{H}}( \cdot ) \), \( {\text{G}}( \cdot ) \) and \( {\text{S}}( \cdot ) \) indicate the color histogram, gradient map and illumination component respectively; n is the number of histogram bins; m is the number of image pixels; w c , w d and w i are three weighting coefficients and we often set them to 1. Obviously, the MSE value in Eq. 6 is inversely proportion to the fidelity of transfer result.

Based on our evaluation metric, we further compare and analyze the performance of our approach with different coefficients: (a) w c = 1, w d = 0, w i = 0; (b) w c = 1, w d = 1, w i = 0; (c) w c = 1, w d = 1, w i = 1. First of all, we prepare ten pairs of source and reference images for color transfer. Secondly, our approach with different coefficients is executed to generate their different transfer results. Thirdly, we calculate their color histograms, gradient maps and illumination components by [2, 22 and 14] respectively. Finally, we measure their ten MSE values to evaluate their performance. As shown in Fig. 3, our approach with the entire color, detail and illumination optimization has the best and stable performance in the objective evaluation.

Fig. 3.
figure 3

Evaluation metric with different coefficients. Note: Compared with the transfer results in the different coefficients, our approach with the entire color, detail and illumination optimization has the best and stable performance in the objective evaluation.

Full size image

5 Experiment Results

In this section, we verify our optimization approach on a variety of images and compare it with the other four approaches (Reinhard et al.’s approach [1], Pitie et al.’s approach [2], Pouli et al.’s approach [3] and Wu et al.’s approach [4]) in the visual appearance. The experiment environment involves a computer with a 3 GHz CPU of Intel Core i5-2400 and 4 GB memory, 64-bit Windows operating system, and Matlab version 7.11. Compared with the other approaches, our approach has lower computation efficiency because of detail and illumination optimization, e.g., we have to take about 20 s for our transfer result in Fig. 1. But we mainly focus on the fidelity of color transfer in this paper.

As shown in Figs. 4, 5, 6 and 7, we provide four groups of source and reference images for comparison, and obtain their corresponding transfer results by the five global example-based approaches. From these figures, we can find that Reinhard et al.’s approach [1] might cause the color overflow and the unharmonious illumination, e.g., the flowerpot in Fig. 4(b), the leaves in Fig. 5(b), the sky in Fig. 6(b) and the sun in Fig. 7(b); Pitie et al.’s approach [2] might produce the extra detail, e.g., the curtain in Fig. 4(c), the tree in Fig. 5(c), the clouds in Fig. 6(c) and the right sky in Fig. 7(c); Pouli et al.’s approach [3] can reshape the histograms but fail to control the detail and illumination, e.g., the whole bathroom in Fig. 4(d), the shadows in Fig. 5(d), the whole sky in 6(d) and the left sky in Fig. 7(d); Wu et al.’s approach [4] can perceive the scene contents but fail to harmonize the original illumination, e.g., the window in Fig. 4(e), the pathlet in Fig. 5(e), the right lighting in Fig. 6(e) and the lake in Fig. 7(e); In contrast, our approach with w c = 1, w d = 1, w i = 1 can effectively eliminate their artifacts and generate a high-fidelity transfer result with color mapping, detail preserving and illumination awareness. Obviously, in Figs. 4(f)–7(f), these experiment results demonstrate that our fidelity optimization model is more satisfying in the visual appearance.

Fig. 4.
figure 4

Example 1 of comparison among color transfer approaches. (a) Source image and reference image. (b) The result by Reinhard et al.’s approach. (c) The result by Pitie et al.’s approach. (d) The result by Pouli et al.’s approach. (e) The result by Wu et al.’s approach. (f) the result by our approach. Note: According to the detail and illumination in source image, our approach can effectively eliminate the existing color, detail and illumination artifacts in the flowerpot part of (b), the curtain part of (c), the whole bathroom of (d), and the window part of (e).

Full size image
Fig. 5.
figure 5

Example 2 of comparison among color transfer approaches. (a) Source image and reference image. (b) The result by Reinhard et al.’s approach. (c) The result by Pitie et al.’s approach. (d) The result by Pouli et al.’s approach. (e) The result by Wu et al.’s approach. (f) The result by our approach. Note: According to the detail and illumination in source image, our approach can effectively eliminate the existing color, detail and illumination artifacts in the leaves part of (b), the tree part of (c), the shadows part of (d), and the pathlet of (e).

Full size image
Fig. 6.
figure 6

Example 3 of comparison among color transfer approaches. (a) Source image and reference image. (b) The result by Reinhard et al.’s approach. (c) The result by Pitie et al.’s approach. (d) The result by Pouli et al.’s approach. (e) The result by Wu et al.’s approach. (f) The result by our approach. Note: According to the detail and illumination in source image, our approach can effectively eliminate the existing color, detail and illumination artifacts in the sky part of (b), the clouds part of (c), the whole sky of (d), and the right lighting part of (e).

Full size image
Fig. 7.
figure 7

Example 4 of comparison among color transfer approaches. (a) Source image and reference image. (b) The result by Reinhard et al.’s approach. (c) The result by Pitie et al.’s approach. (d) The result by Pouli et al.’s approach. (e) The result by Wu et al.’s approach. (f) The result by our approach. Note: According to the detail and illumination in source image, our approach can effectively eliminate the existing color, detail and illumination artifacts in the sun part of (b), the right sky part of (c), the left sky part of (d), and the lake part of (e).

Full size image

6 Conclusion and Discussion

In this paper, we present a new fidelity optimization model for color transfer. Most importantly, to improve the fidelity in terms of color, detail and illumination, we formalize our approach as an optimization problem with three energy functions: color mapping, detail preserving and illumination awareness. Moreover, we also propose an objective evaluation metric to measure the fidelity of our approach in the different coefficients. The final experiment results verify our optimization approach’s effectiveness for the high-fidelity color transfer.

Although our approach can yield a high-fidelity transfer result, it must take more computing resources in time and memory for constructing the illumination affinity matrix and solving a large and sparse linear system. Thus users should weigh the efficiency and fidelity to choose the other approaches or ours. In the future, we plan to refine the construction of illumination affinity and introduce some acceleration techniques [23–25] for real-time solving. Moreover, based on space-time consistency, we will also extend our fidelity optimization model to process the video streams.