1 Introduction

As a traditional art in China, Chinese painting differs from other art in its expressive brush strokes and ink diffusion. To ideally render water-and-ink painting, many researchers attempted to use computer simulation for such complicated texture generation [13, 15]. In this paper, we aim to render Chinese painting with other artistic style, which is regarded as a style transfer problem.

Style transfer is to synthesize an image that combines the structure of a original image with the artistic style of the example image. In this work, it is a process of migrating a style from an example image to Chinese painting, which can be generally regarded as transferring two different painting style. In animation production and video post-production fields, style transfer and related approaches are highly interested as they facilitate generating different scenes [9, 12]. Although various methods have been proposed for this issue, style transfer task has not been well-defined. The core difficulty is how to distinguish style feature from semantic content in an image, including all visual attributes such as texture, strokes, color and shading.

Previous study offers two distinct methods for style transfer: One is generalization of classic texture synthesis approaches, such as the works in [2, 3], in which optimal patches of a single image are expected to be found based on local similarity. An alternative technique for style transfer problem emerged in recent years, defining content and style representation of two images and using Convolutional Neural Networks (CNN) to merge the corresponding content and style [7].

Our work is motivated by patch-based texture synthesis approaches. In spite of traditional patch-based texture synthesis methods made an impressive success for style transfer, the limitations should be overcome. For example, the local texture synthesis is accomplished in the same and fixed size patches throughout the whole image, where the size of the patch is a tradeoff between the style and the content to be preserved in the output image. The size of patch should be large enough to exhibit the patterns that characterize the example style, yet small enough to reconstruct the realistic content of original image. Another limitation is that traditional constrains in transferring consider only luminance and local neighboring similarity of target image, without color information. Hence, we propose a style transfer method for Chinese painting which is able to overcome the limitations. The main contributions of this work are summarized as follows:

  • We adaptively divide target Chinese painting into patches based on its local similarity for texture synthesis, in stead of using patches of constant size, so as to achieve a realistic reconstruction of the original image while present most noticeable example style;

  • Constraints are modified in the process of texture synthesis, where color is considered as a relevant factor guiding local texture transfer. It may guarantee the validity in transfer process, where the futile texture is prevented.

2 Related Works on Style Transfer

Style transfer can be considered as a special case of texture synthesis, where the content image influences the regular synthesis process. In the literatures of traditional texture synthesis and transfer, example-based methods are to generate a texture image by computing non-parametric sampling from a given example style image based on Markov Random Field (MRF). One of the earliest works in [2] by Efros and Leung takes a pixel to be synthesized by random sampling from a set of candidate pixels that are selected from an example texture image. This process is repeated for every output pixel by growing from the initial region until all the output pixels have been already synthesized. Intuitively, the neighborhood size should be equal to the texture element sizes. Otherwise, the output texture may be too random or regular pattern may be reduced. The quality and speed of these pixel-based approaches [2, 14] were improved by path-based one. In [3], a patch-quilting procedure for texture synthesis is proposed, and then extended it to texture transfer. Patch-based texture transfer is similar to pixel-based one, except that instead of synthesizing pixels, it copies patches.

The work in [8] suggested texture optimization as texture synthesis method beyond pixel-based and patch-based algorithms. The algorithm synthesizes an output texture in the units of pixels, but unlike previous pixel-based methods that synthesize pixels one by one in a greedy fashion, this technique considered all pixels together, determining pixel values by minimizing a quadratic energy function. This energy function has been modified by the latest work in [4] to match the transfer task better. In details, both content and example style image were restricted by a segmentation mask adding to the energy function, in order to determine which parts to be transferred and preserved.

Recently, an impressive work of style transfer is using Convolutional Neural Networks (CNN)[7]. Their methods adopt a pre-tranined CNN to extract features from both the style and the content images, respectively.

Motivated by [5], which consider an explicit probability density modeling of the problem and computes an approximate Maximum a Posteriori(MAP) solution based on an iterative optimization of Belief Propagation or Graph cuts, we propose a novel style transfer method for Chinese painting. Unlike the traditional patch-based algorithm in [3], we propose an adaptive patch for style transfer. Especially given that our target image in this work is black-and-white Chinese painting with expressive content, we improve classic style transfer algorithms by modifying the optimal match condition to overcome such a challenge.

3 Problem Description

Traditionally, Chinese painting (water-and-ink) is presented by ink diffusion of different degree on the Xuan paper. The objects are in a wide range of scale, painted by complex and expressive brush strokes. In other words, while some scene objects are always painted with rough brushwork, the key objects are painted in detail with subtle brushwork. For example, in Fig. 1(a), the distant mountains are roughly painted by great water-and-ink diffusion but the fisherman and the texture of the mountains nearby are exhibited subtly by slight ink spreading. Moreover, ink diffusion can be also used for rendering Chinese painting as “color”, such as the representation of cloud and shading.

Our goal is to transfer other artistic styles such as impressionism and post-impressionism to Chinese painting. Consequently, we propose a style transfer method that adopts an adaptive patch for patch-based texture transfer, and colorization to guide the process of style transfer. At first, we give the problem definition of style transfer for Chinese painting.

Given a Chinese painting \(C: \varOmega _{C}\in \mathbb {R}^{3}\), and a style image \(S: \varOmega _{S}\in \mathbb {R}^{3}\) with certain style. We aim to synthesis an image \(C_{out}\) which captures the style of S while preserves the semantic content of C. This can be considered as finding a mapping \(f: \varOmega _{C}\rightarrow \varOmega _{S}\) which comfirms each element \(X\in \varOmega _{C}\) with a corresponding element \(Y=f(X)\in \varOmega _{S}\).

Applying a similar idea for patch-based texture transfer, the correspondence mapping f should be a piecewise constant translation mapping on region \(P=\{P_{i}\}_{i=1}^{n}\) of \(\varOmega _{C}\). In order to extract the style feature of S while preserving the structure of C, the region P should be obtained based on the painting elements of C, and the texture as well as color of S should be taken into account for the optimal corresponding f(x). Especially, to transfer the style elegantly, smoothness is required on the boundary between neighboring regions.

4 Style Transfer for Chinese Painting

In this section we detail the proposed style transfer algorithm. In order to meet the requirements mentioned above, our approach can divided into three main steps:

  • Adaptively decompose \(\varOmega _{C}\) into n regions P;

  • Locally render \(\varOmega _{C}\) according to the color of S;

  • Find the optimal mapping f based on MRF model;

Moreover, corresponding experiment results are presented to illustrate the performance of each step. We note that our style transfer is accomplished in YUV color space, since we consider both luminance and chrominance in the process of texture transfer.

Fig. 1.
figure 1

Illustration of adaptive decomposition (Color figure online)

Full size image
Fig. 2.
figure 2

Comparison of adaptive patches and fixed-size patches: Our method with adaptive patches and the minimal size of patches is \(4\times 4\) as showed in (c); (d) present the result of Image Quilting algorithm with patches of fixed size \(8\times 8\). (Color figure online)

Full size image

4.1 Adaptive Decomposition for Chinese Painting

We firstly recall that in patch-based texture transfer, the original image to be rendered is decomposed into fixed size patches, and assign one node of a Markov network. Generally, if the size of patches are small (for example the size of 8\(\times 8\)), the content of original image can be ideally reconstructed yet the style of the example style image is nearly obvious; on the contrary, if large patches have been chosen for texture synthesis, the considerable details of original image are lost. To reconstruct the realistic content of original Chinese painting while inheriting the example style, we divide the original image into adaptive-size patches based on its structure and pixel distribution.

Let decomposition starts with one single region \(P_{i}\in \varOmega _{C}\), of size \(m\times m\). Each region \(P_{i}\) is divided into four equal squares, with each size of \(\frac{m}{2}\times \frac{m}{2}\), if pixel value \(X_{i}=(x_{1},x_{2},\ldots x_{m\times m})\in P_{i}\) satisfies:

$$\begin{aligned} D(X_{i})>\sigma \ \mathrm{or}\ \ m > \omega \end{aligned}$$
(1)

where \(\sigma \) is the threshold; \(D(X_{i})=(\max (X_{i})-\min (X_{i}))\) presents the difference between the maximum and minimum value in region \(P_{i}\), and \(\omega \) is the maximum patch size allowed in the quadtree.

The local variance of a quadtree cell decides whether a cell is divided into four cells, which depends the details in C. As illustrated in Fig. 1(b), the more delicate elements in the original image are divided into the more smaller patches to be transferred, such as the trees and fisherman nearby. Thus, the content of original image can be perfectly preserved in texture synthesis, while the style feature can be reflected as much as possible, as showed in Fig. 1(d). Obviously, our decomposition only depends on the structure of original image, rather than the stopping criteria for quadtree splitting in [6].

In Fig. 2, we present a comparison of between our adaptive patches and Image Quilting with fixed-size patches in [3]. To make it clear, We choose the smallest size of patch allowed in two algorithms, and highlight two specific differences in the results by red rectangles. It can be observed that two persons on the bridge and the curved roof of pavilion reconstructed by our method are more clearer than those reconstructed by Image Quilting as showed in Fig. 2(c) and Fig. 2(d). These results present that our adaptive patches preserves the original content better than fixed-size patches.

Fig. 3.
figure 3

Illustration of locally color transfer: Our result is more reasonable than the one of Image Quilting method, since there is futile texture on the farmland by Image Quilting [3]. And the color gamut of our result is more similar to the color gamut of style image compared to the results of Split and Match method [6]. (Color figure online)

Full size image

4.2 Locally Color Transfer

Color style transfer is an essential step in style transfer which has usually been done separately after texture transfer in classic approaches. Due to that the brightness and darkness in Chinese painting are exhibited by complex ink diffusion, the colors are usually gray, black and white, while the other artistic style is generally colorful. Without chrominance information, the color fidelity of example style cannot be guaranteed during reconstructing \(C_{out}\). It is worse that the futile texture may appear which is not conform to semantic content of the original image. Thus, instead of transferring texture only in luminance, we consider the chrominance information.

Here, we preprocess colorization for original Chinese painting before texture transfer. Specific colors in S are extracted as color seeds for local rendering through colorization method suggested in [10]. Then, the rendered image \(\tilde{C}\) guides the texture transfer as one of criteria in chrominance. In detail, we search for the optimal match for texture transfer in luminance as well as chrominance (in YUV color space), which is described in next 4.3.

Similarly, we show the transfer results in Fig. 3. It is noted that if consider luminance as the only matching condition for texture transfer, the futile texture are synthesized, as showed in Fig. 3(d). From semantic understanding, the color of farmland should be yellow or green, but Image Quilting algorithm (and other classic methods that only consider luminance) synthesizes blue and white texture. As presented in Fig. 3(c), compared with the traditional algorithms, our method can obtain a reasonable output image since the chrominance is considered. In addition, the color gamut of our result is more similar to the color gamut of style image than the results of Split and Match method shown in 3(e). It is indicated that the color style can be better extracted with chromatism information.

Fig. 4.
figure 4

Markov network for our work: Each node in the network describes a local adaptive patch of original or example image.

Full size image
Fig. 5.
figure 5

Transfer results on different style examples: Original Chinese painting (left column), example style images (middle column), and the style transfer result (right column).

Full size image

4.3 Optimal Match

As mentioned above, both the original image and example style image are divided into patches where each patch is one node of a Markov network. With the framework of Markov Random Field (MRF), the problem of patch-based style transfer can be solved through computing the Maximum Posteriori from a well chosen joint probability distribution on all patches [5]. Thus, the optimal mapping f can finally be found with MRF model.

The MRF model in our work is illustrated in Fig. 4, which can be found that the links on original image connect adaptive patches rather than fixed size patches. We search for the optimal match for each patch by finding maximum a posteriori (MAP), which is equally maximizing the joint probability over the \(X_{i}\) and \(Y_{i}\), that can be written as

$$\begin{aligned} Pr(X_{1}, X_{2},\ldots X_{N}, Y_{1}, Y_{2},\ldots Y_{N}) = \prod _{(i,j)\in N}\varPsi _{i,j}(X_{i}, X_{j})\prod _{k\in N}\varPhi _{k}(X_{k},Y_{k}), \end{aligned}$$
(2)

where \(\varPsi _{i,j}(X_{i}, X_{j})\) are pairwise interaction potentials between neighboring nodes i and j, while N(i, j) denotes the neighbors of patches. \(\varPsi _{i,j}(X_{i}, X_{j})\) ensures that neighboring patches are similar in their overlapping region and it can be written as

$$\begin{aligned} \varPsi _{i,j}(X_{i}, X_{j})= \exp (-E({X}_{i},{X}_{j})) \end{aligned}$$
(3)

where \(E({X}_{i},{X}_{j})=\Vert {X}_{i}-{X}_{j}\Vert ^{2}\) is the error term of the overlapping region between two patches. \(\varPhi _{k}(X_{k},Y_{k})\) are the data penalty functions given by

$$\begin{aligned} \varPhi _{k}(X_{k},Y_{k})=\exp (-\theta (X_{k},Y_{k})). \end{aligned}$$
(4)

where \(\theta \) is the weighted error term between the newly chosen block and the old blocks. As discussed in 3.3, we use colorized image \(\tilde{C}\) to guide the texture transfer, hence, \(\theta [X_{k},Y_{k}]\) is defined as

$$\begin{aligned} \theta (X_{k},Y_{k})=\alpha d({X}_{k},{Y}_{k})_{Ori}+\beta d(X_{k},Y_{k})_{Ch}+\mu d(X_{k},Y_{k})_{L}. \end{aligned}$$
(5)

We modify the criterion in [3] by adding \(d(X_{k},Y_{k})_{Ch}\), presenting the square error of patches between rendered image \(\tilde{C}\) and example style image S. \(d({X}_{k},{Y}_{k})_{Ori}\) is the square error of the overlapping regions in the original image C, and \(d(X_{k},Y_{k})_{L}\) is the square error term of patches between original image and style image in luminance. \(\alpha \), \(\beta \) and \(\mu \) are three positive weights that no bigger than 1 (respectively fixed to 0.2, 0.2 and 0.6 in all experiments).

Finally, we achieve an optimal boundary of adjacent patches to remove visibly artificial seams. This minimal cost path through the overlap region can be done with dynamic programming [1]. Other transfer results with respect to different example style are presented in Fig. 5. With different style, our algorithm is able to transfer example style while ideally reconstruct the content of the original painting.

Fig. 6.
figure 6

Comparison with CNN approach: Original Chinese paintings (first column), different style images (second column), our results (third column), and results of CNN approach (last column). (Color figure online)

Full size image

5 Comparison of Our Method and Other Approaches

In this section, we would like to make a comparison between our method and recent style transfer approaches.

As shown in Fig. 6, we present the experimental results with our method and a popular method Convolutional Neural Network (CNN) with the parameter setting in [7]. Both our method and CNN achieve ideal reconstruction for original content. The subtle texture feature of the style images can be captured with our method such as the wavy strokes in Van Gogh’s Starry night. Even the detail texture element like the yellow and white points are preserved in our result, which hardly appear in CNN transfer results. And the color gamut of our results is more closer to the color gamut of style images, compared with the results of CNN. This is due to that in the style transfer process, we choose the optimal patches in the original style image as the generated patches in stead of extracting the abstract style feature. While CNN uses deep and abstract style representation, it loses low-level pixel features of the style image. Moreover, CNN has the trade-off problem of style and content matching, which has been mentioned in [7]. Similarly, the transfer method in [11] applies MRF prior defining the loss function for CNN to control the abstract style layout yet our algorithm improves compatibility functions of MRF to generates style directly from the style image, rather than extracting the abstract style step by step.

As we mentioned in Sect. 4.1, our adaptive decomposition for the content image only depends on the local variance, while the recent work by Frigo et al., in [6] also regards the similarity between the content image and the style image as the decomposition criterion. Most importantly, compare with Split and Match method transfers color style separately after texture transfer, we combine texture transfer and color style transfer, by guiding the texture transfer process with chromatism information. As depicted in Fig. 3, our result maintains the original color style of the style image including green, blue and yellow color. Yet the results of Split and Match method almost miss yellow color feature. The color gamut of our result is more closer to the color gamut of style image.

6 Conclusion

In this paper, we regard the rendering problem of Chinese painting as a style transfer issue and propose a new style transfer method for Chinese painting. Based on the characters of Chinese painting where the painting elements are always have obviously distinct scale, adaptive-size patches are applied for texture transfer in our approach. Additionally, we modify the constraints in texture transfer based on MRF model, considering color information of both style image and colorized original image. The local colors of style image are extracted as color seeds for rendering the black-and-white Chinese painting, which helps to guide the process of texture transfer.

The experimental results of each step are presented to clearly illustrate the improvement by our proposed algorithm. The results suggest that decomposing target Chinese painting with adaptive patches to be transferred is able to well preserve the original content while transferring example style, and the color style can be captured with chromatism information. Finally, we discuss the comparison of our method and other state-of-the-art style transfer methods, including patch-based approach and CNN framework.