Interactive Artistic Multi-style Transfer

Artistic style transfer is to render an image in the style of another image, which is a challenge problem in both image processing and arts. Deep neural networks are adopted to artistic style transfer and achieve remarkable success, such as AdaIN (adaptive instance normalization), WCT (whitening and coloring transforms), MST (multimodal style transfer), and SEMST (structure-emphasized multimodal style transfer). These algorithms modify the content image as a whole using only one style and one algorithm, which is easy to cause the foreground and background to be blurred together. In this paper, an iterative artistic multi-style transfer system is built to edit the image with multiple styles by flexible user interaction. First, a subjective evaluation experiment with art professionals is conducted to build an open evaluation framework for style transfer, including the universal evaluation questions and personalized answers for ten typical artistic styles. Then, we propose the interactive artistic multi-style transfer system, in which an interactive image crop tool is designed to cut a content image into several parts. For each part, users select a style image and an algorithm from AdaIN, WCT, MST, and SEMST by referring to the characteristics of styles and algorithms summarized by the evaluation experiments. To obtain richer results, the system provides a semantic-based parameter adjustment mode and the function of preserving colors of content image. Finally, case studies show the effectiveness and flexibility of the system.


Introduction
Artistic style transfer is to render an image (called content image) in the style of another image (called style image), which is a challenge problem in both image processing and arts. Deep neural networks are adopted to artistic style transfer and achieve remarkable success. Deep neural networks can encode the content information of an image with low layers and extract the style information from high layers.
The transfer method combines the content and style of arbitrary images. In recent years, there are several popular DNN-based style transfer algorithms, such as AdaIN (adaptive instance normalization), WCT (whitening and coloring transforms), MST (multimodal style transfer), and SEMST (structure-emphasized multimodal style transfer). However, these algorithms modify the content image as a whole using only one style and one algorithm, which is easy to cause the foreground and background to be blurred together. In fact, users may want to modify the foreground of an image with strong artistic characteristics and the background with weak artistic characteristics to highlight areas of interest and avoid blurring together.
In this paper, an iterative artistic multi-style transfer system is built to combine DNN-based style transfer algorithms and realize multiple style transfer in an image. First, we conduct a subjective evaluation experiments with art professionals to evaluate transfer algorithms in more details from the perspective of art. Then, based on the evaluation results, we propose an iterative artistic multi-style transfer system. An interactive image crop tool is designed to cut a content image into several parts. For each part, a style image and an algorithm are assigned through user selection. Ten typical artistic styles such as Van Gogh's style, Vienna Secession are provided by default. Besides, users can upload images as artistic style references. The system provides four popular and fast style transfer algorithms, which are AdaIN, WCT, MST, and SEMST. To obtain richer results, the system provides a semantic-based parameter adjustment mode and the function of preserving colors of content image. The case studies show that the iterative artistic multi-style transfer system can obtain harmonious and diverse results with different style combinations.
The main contributions include: • An open framework for fine-grained evaluation of transfer algorithms from the perspective of art is formulated. The framework is summarized by subjective experiments of art professionals. It includes six universal evaluation questions, which can be applied to arbitrary artistic styles. It also contains personalized answers for ten typical artistic styles and allows adding more artistic styles by following the framework. • An artistic multi-style transfer system with flexible user interactions is built. An interactive image crop tool is designed for user-defined segmentation. In addition to providing ten representative styles and four different algorithms, the characteristics of these styles and algorithms are integrated into the system to provide professional guidance to users.
The rest of the paper is organized as follows: Sect. 2 gives related work. Section 3 describes the subjective evaluations on artistic transfer algorithms. Section 4 proposes the interactive artistic multi-style transfer system. Section 5 evaluates the system by case studies. Section 6 finally draws the conclusions.

Artistic Transfer Algorithm
Early Artistic Transfer Algorithms. The universal artistic style transfer algorithms aim to transfer arbitrary visual styles to content images. Early transfer algorithms are related to color transfer [21] and texture synthesis [6]. Some image processing methods are adopted, such as non-parametric sampling [7], texture matching [13], and edit propagation [18]. Deep Neural Network-Based Transfer Algorithms. With the development of deep neural networks, they are applied to artistic style transfer and achieves good improvement. Gatys et al. proposed a convolutional neural networkbased artistic transfer algorithm, which found that higher layers of the network represent the style, and lower layers represent the content [10]. The algorithm uses neural representations to separate and recombine content and style of arbitrary images. However, the iterative optimization process of this algorithm is slow.
Faster Transfer Algorithms. Huang and Belongie proposed an adaptive instance normalization (AdaIN) layer based on VGG-19 network [22] that aligns the mean and variance of the content features with those of the style features to achieve speed [15]. Li et al. formulated style transfer as an image reconstruction process coupled with feature transformation, that is whitening and coloring (WCT) [17]. Zhang et al. proposed a multimodal style transfer (MST) method, which explicitly considers the matching of semantic patterns in local content features and substyle components under a graph cut formulation [24]. Inspired by MST, Chen proposed a structure-emphasized multimodal style transfer (SEMST), which extract structural information to automatically determine the optimal number of clusters [4]. More neural style transfer methods are available in the survey [16].
Additional Functions of the Transfer Algorithms. Some additional functions are added based on the universal style transfer algorithms, such as the color preservation [11], color transfer [8], and user control [9]. The data brush is an interactive web application which enables users to paint a style onto selected areas of an image [9]. An unpaired image-to-image translation method is proposed [25].
The above algorithms are the universal style transfer algorithms, which proposed a single network to transfer multiple styles. In fact, the characteristics of different styles vary greatly and require different algorithms to transfer. In this paper, we will compare the above four faster universal transfer algorithms by user studies, which are AdaIN, WCT, MST, and SEMST. We summarize the user needs and design the interactive artistic multi-style transfer system to choose the proper styles for different contents.

Evaluation Standards and Methods
The evaluation methods are divided into qualitative ones and quantitative ones. User studies are usually adopted for the qualitative assessment of artistic style transfer algorithms. Some examples based on different transfer algorithms are presented in the paper [17,22]. Li et al. evaluated different transfer algorithms by voting her/his ONE favorite result for each style [15,17]. For quantitative evaluation, the content and style loss of synthesized images and speed are often used as evaluation standards [22].
The standards of these evaluation methods are highly subjective, and the evaluation results are highly dependent on the participants. Everyone has a unique definition of the favorite result. It is unreasonable to use the same evaluation standard for different styles. For example, for the style of the abstract painting, the result image should retain as little original content as possible, while for oil paintings and watercolors, the result image should keep as much original content as possible. Therefore, in this paper, we have indepth interviews with art professionals to formulate different evaluation standards for different styles and types of content images. These evaluation standards are quantified as much as possible into a consensus between different people.

Evaluation on Artistic Transfer Algorithms
The overall pipeline of our framework is shown in Fig. 1. First, we invited five art professionals for in-depth interviews. Inspired by the art theory [23], analysis of art masterpieces [20], theory of color matching [5,12], and analysis of artistic styles [14,19], an evaluation framework from the perspective of art is formulated in this paper, including the universal evaluation questions and personalized answers for ten typical artistic styles. The evaluation questions are set to compare the transfer results with the style images and the content images, which can be applied to arbitrary artistic styles. It is an open framework that can add personalized answers for more artistic styles.

Evaluation Standards
Ten typical style types are selected, which are oil painting, ink wash painting, lacquer painting, watercolor painting, ukiyo-e, printmaking, abstract painting, symbolism, sketch, and illustration. A typical image for each style is also selected. For example, Van Gogh's famous "Starry Night" is the typical image for painting, Xueshi Bai's "LI JIANG YU GUI" is taken as an example for ink wash painting, and Gustav Klimt's painting "Portrait of Adele Bloch-Bauer" is selected for symbolism. Figure 1 shows all the typical images for the ten styles.
The five art professionals summarized the evaluation questions for each artistic style, 13 questions on average. Some questions analyze the similar characteristics from different perspectives and can be merged. For example, the question "whether the result image uses the same material as the style painting" and "whether the result image has the same texture as the style painting" describe the textures that different materials bring to people. Besides, we use the creation of details to include the line processing, applications of brush strokes, etc. Finally, we categorize them into six universal problems from two perspectives of content and style in the result image. These questions are summarized as follows: • Q1. The degree of refinement and generalization of the content image by the transfer algorithms, that is how much of the content in the original content image is retained. The higher the degree of the refinement and generalization, the less the content of the original content image is retained. Participants label the transfer result as high (H), medium (M), and low (L). • Q2. The degree of the same texture as the style painting. • Q3. The degree of the same feeling as the style painting. • Q4. The degree of the same tone used as the style painting. • Q5. The degree of the color contrast. The personalized answers of the above questions for ten artistic styles are summarized in Table 1. For each question, participants label the transfer result as high (H), medium (M), and low (L).

User Study
We conducted a user study to evaluate the four methods, which are AdaIN, WCT, MST, and SEMST. In the experiment, there were 60 trials (Fig. 3). For each trial, four compared results were displayed on the screen in random order with the content image and the style image at the same time, as shown in Fig. 4. The names of the transfer algorithms were removed. Each participant was asked to answer six questions for each trial. The four transfer results are labeled separately for each question. Therefore, in each trial, the participant gave 24 labels. In total, 1440 labels should be marked, which requires artistic knowledge and is a huge workload. Finally, we invited ten art professionals to participate in the user study.
We summarized the label results of user studies. Some of the conclusions by the result analysis are as follows: • Different algorithms should be provided in the multi-style transfer. In terms of algorithm performance, the results transferred by WCT are more general and freehand and make the "painting taste" more intense than the other three algorithms. However, WCT is too refined in dealing with complex details and make the results retain less content and lack of recognition. Therefore, WCT is good at processing content images with less details and abstract painting as target style. AdaIN performs best in the sketch style. MST and SEMST algorithms can preserve more content information. MST performs best in the lacquer painting style. SEMST performs best in the printmaking style. • Different parts of an image should be set diverse styles to match their different semantics. In terms of image

Interactive Artistic Multi-style Transfer
We build the artistic multi-style transfer system, which enables users to edit the content image with multiple styles by flexible user interaction. The interface of the system, shown in Fig. 5, includes five functional areas, which are artistic style selection, display area, tool area, algorithm selection, and part selection. First, the user uploads the content image and cuts it into parts through the user-defined image cropping tools. The summarized characteristics of styles and algorithms are provided for selecting a style image and an algorithm for each part. Finally, the system achieves style transfer and image blending.

User-Defined Image
Cropping. An interactive image crop tool is designed to freely cut a content image into several parts by users. The content image in Fig. 5 is from Pexels [2]. The five buttons in the tool area from left to right at the bottom of the interface are are used to upload images, select line, select curve, switch display states, and download images. When opening the interface for the first time, there is a big plus sign in the display area. Click the plus sign to upload a content image and it is displayed in the middle of the interface. The upload button in the tool area is provided to upload a new image. Users can draw a closed area by lines and curves on the content image by the line selection tool and the curve selection tool, respectively. The userdefined areas are shown in the part selection area. There is a default part that is the whole image at first, then is the area not selected after cropping. Users can add a new part and delete existing parts in the part selection area. The display switching button is used to switch two display states: only showing the currently processed image and simultaneously showing the currently processed image and the original content image. If you are satisfied with the result image, you can click the download button to save it.
Artistic Style Selection. In the area of artistic styles on the left side of the interface, ten typical style types are provided by default, which are oil painting, ink wash painting, lacquer painting, watercolor painting, ukiyo-e, printmaking, abstract painting, symbolism, sketch, and illustration. Each style is presented by an image and a brief text introduction. For example, the typical painting "Surfing the Great Wave off Kanagawa" of the ukiyo-e style has vivid colors with blue, white and tan, clear and Fig. 4 One trial in the user study. Four compared transfer results were displayed on the screen in random order with the content image and the style image at the same time. And the names of the transfer algorithms were removed refined lines full of tension, rich layers, and asymmetric composition, which bring a powerful visual impact. Nonprofessional users refer to these instructions to choose styles and upload style images. Besides, users can upload any images as artistic style reference images. There is a "favorite" button in the upper right corner of this area to collect style images uploaded by users.
Algorithm Selection and User-Defined Multi-style Transfer. Four transfer algorithms are provided, which are AdaIN, WCT, MST, and SEMST. They have different emphasis on the tradeoff between content and style and can generate rich transfer results. Besides, the system provides a semantic-based parameter adjustment mode and the function of preserving colors of the original content image. They are shown in the expansion panel when selecting a style image. For the semantic-based parameter adjustment mode, a slide is adopted to set the proportion of content and style in the result image, which corresponds to the weights in the deep neural networks. Since the transfer algorithms are easy to cause color mixing in the result image, the function of preserving colors is provided for the case where the saturation of the content image is high. In terms of user interaction, users click each part in the part selection area and select a style image and an algorithm by referring to the summarized characteristics of styles and algorithms.

Evaluation on the Multi-style Transfer System
To evaluate the multi-style transfer system, the case studies are adopted. We invited participants to use the system and some results are shown in Figs. 6 and 7. The context images in Figs. 6 and 7 are from Pexels [2]. In Fig. 6, a content image is cut into several parts in two ways. Each part is assigned a target style image. The style images are arranged from top to bottom in the order of the number of the segments. On the top of Fig. 6, doors are the first part, the upper window is the second part, the interior is the third part, and the other unselected areas are the fourth part. We can see that the transferred result for each part shows the characteristics of corresponding style image while preserving the information of the content image. Furthermore, the result of the combination looks harmonious and beautiful. In Fig. 7, the style images at the bottom of each result are arranged from left to right in the order of the number of the segments. The harmonious results with the combination of different styles show the diversity of the system.
To evaluate the effectiveness of the system, we organized a user study. 74 Participants were invited with ages   (2) the naturalness and comfort level of the color combination. The Likert Scale is adopted, where "1" means very discordant or uncomfortable, "3" means normal, and "5" means very harmonious or comfortable.
The statistical results of the user study are shown in Fig. 8. It shows the average scores for the 14 transfer results following the order from top to bottom and left to right in Figs. 6 and 7. The overall average harmony score of the style fusion is 3.6 and the average naturalness score of the color combination is 3.5. Since the evaluation for the artistic effects are very subjective, the overall score greater than 3 is satisfactory, showing the effectiveness of the system, which can assist users to generate diverse, harmonious, and beautiful transferred results with flexible user interactions.

Evaluation on the Color Preservation
The transfer algorithms are easy to cause color mixing in the result image. If the two parts of the content image are with the similar color distribution, while the colors of the two style images are very different, the color combination of the transfer result may be discordant. We invited 74 participants with ages from 20 to 24, each of which had normal vision. Figure 9 shows two test cases. Participants were given a test case including the content image, the image segmentation, the style images, the transfer result without color preservation, and the transfer result with color preservation. Participants were asked to rate the two transfer images by the naturalness and comfort level of the color combination in Likert Scale, where "1" means very uncomfortable, "3" means normal, and "5" means very comfortable.
The statistical results of the user study are shown in Fig. 10. The average scores of (c) and (d) in the above case in Fig. 9 are 3.3 and 3.8, respectively. The average scores of (c) and (d) in the below case are 2.7 and 3.8, respectively. In the above case, the color combinations of the style images are all cool, so the transfer result without color preservation is still acceptable. However, when the color combinations of   Fig. 9, the transfer result without color preservation is very discordant and uncomfortable. Therefore, the color preservation function in the system is convenient for users to boldly choose the style images without considering the disharmony caused by their different color combinations.

Evaluation on the Semantic-Based Parameter Adjustment Mode
Different artistic styles have different texture strengths caused by diverse brush strokes. When the textures in some areas of the multi-style transfer result are too strong, the semantic-based parameters can be adjusted to make the transfer result more harmonious. The cases showing the advantages of the semantic-based parameter adjustment mode are shown in Fig. 11. We organized a user study and the experiment settings are the same as the evaluation on the color preservation above. The two cases in Fig. 11 are the test materials. The criteria are the harmony of the textures of the transfer result. The statistical results of the user study are shown in Fig. 12. The average scores of (c) and (d) in the above case in Fig. 12 are 3.0 and 4.0, respectively. The average scores of (c) and (d) in the below case are 3.2 and 3.9, respectively. The user study shows that the semantic-based parameter  result with the semantic-based parameter = 1 for all segments, and d transfer result. 1 = 0.7 , 2 = 0.5 , 3 = 0.5 , and 4 = 0.3 for segments in the above case. 1 = 1 , 2 = 0.5 , 3 = 0.5 , and 4 = 0.5 for segments in the below case adjustment mode can help to unify the harmony of the textures of the transfer result. Taking the above case in Fig. 12 as an example, the textures of printmaking are strong, so the semantic-based parameter 4 is set smaller to match with other segments.

Conclusion
We present an artistic multi-style transfer system with flexible user interactions. The system is based on the open framework for fine-grained evaluation of transfer algorithms after subjective experiments of art professionals. The framework includes six universal evaluation questions for arbitrary artistic styles and personalized answers for ten typical artistic styles. Besides, it allows adding more artistic styles by following the framework. The system provides an interactive image crop tool for user-defined segmentation, representative styles, and different algorithms for diverse result generation. The case studies validate the effectiveness and flexibility of the system.