Computational Decomposition of Style for Controllable and Enhanced Style Transfer

Abstract

Neural style transfer has been demonstrated to be powerful in creating artistic images with help of Convolutional Neural Networks (CNN), but continuously controllable transfer is still a challenging task. This paper provides a computational decomposition of the style into basic factors, which aim to be factorized, interpretable representations of the artistic styles. We propose to decompose the style by not only spectrum based methods including Fast Fourier Transform and Discrete Cosine Transform, but also latent variable models such as Principal Component Analysis, Independent Component Analysis, and so on. Such decomposition induces various ways of controlling the style factors to generate enhanced, diversified styled images. We mix or intervene the style basis from more than one styles so that compound style or new style could be generated to produce styled images. To implement our method, we derive a simple, effective computational module, which can be embedded into state-of-the-art style transfer algorithms. Experiments demonstrate the effectiveness of our method on not only painting style transfer but also other possible applications such as picture-to-sketch problems.

Appendices

Appendix 1: The Stylization Effect of Every Activation Layer in VGG19

Since different layers in VGG19 [18] represent different abstract levels, we experiment the stylization effect of every activation layer on a couple of different style images as are shown in Figs. 9 and 10. From our experiments, it can be noticed that not every layer is effective in style transfer and among those that work, shallow layers only transfer the coarse scale of style (color) while deep layers can transfer both the coarse scale (color) and detailed scale (stroke) of style, which conforms to the result of scale control in [9]. Since ‘relu4_1’ performs the best in style transfer after same learning epochs, we determine to study the feature map of ‘relu4_1’ in our research.

We further visualize each channel of the feature map of the style image using t-SNE [4], as are shown in Figs. 11 and 12 where the similarity of both the results is quite interesting. However, we could not explain those specific patterns shown in the visualization results yet. What’s more, the relationship between the similarity of the visualization results with the similarity of the stylization effects of different VGG layers remains further study as well.

Fig. 10.

Styled images using [8] with same epochs using every single activation layer from the pre-trained VGG19.

Fig. 11.

Low dimensional projection of all channels of every single layer of the style image (Fig. 9(b)) via t-SNE [4].

Fig. 12.

Low dimensional projection of all channels of every single layer of style image (Fig. 10(b)) via t-SNE [4].

Fig. 13.

(a) Chinese paintings; (b) Oil paintings (by Leonid Afremov); (c) Pen sketches; (d) low-dimensional projections of the spectrum of style (a–c) via Isomap [20].

Fig. 14.

Low-dimensional projections of the spectrum of large scale of style images via Isomap [20]. The size of each image shown above does not indicate any other information, but is set to prevent the overlap of the images only.

Appendix 2: The Manifold of Spectrum Based Methods

We analyze the spectrum space by projecting the style bases via Isomap [20] into low dimensional space where the X-axis represents the color basis and the Y-axis represents the stroke basis, which can analytically demonstrate the effectiveness and robustness of spectrum based methods. Three artistic styles are experimented (shown in Fig. 13(a–c)). Chinese paintings and pen sketches share similar color style which is sharply distinguished with oil paintings’ while the stroke of three artistic styles are quite different from each other. Thus, as in shown in Fig. 13(d), Chinese paintings and pen sketches are close to each other and both stay away from oil paintings in X-axis which represents color while three styles are respectively separable in Y-axis which represents stroke, which completely satisfies our analysis of the three artistic styles.

When we apply the same method to large scale of style images (Fig. 14), X-axis clearly represents the linear transition from dull-colored to rich-colored. However, we fail to conclude any notable linear transition for Y-axis from the 2-dimensional visualization probably because it is hard to describe the style of stroke (boldface, length, curvity, etc.) using only one dimension.

Fig. 15.

The styled image with stroke basis intervened using spectrum based methods. The left most row shows the style images (From top to bottom: The Great Wave off Kanagawa - Katsushika Hokusai; Composition - Alberto Magnelli; Dancer - Ernst Ludwig Kirchner; Pistachio Tree in the Courtyard of the Chateau Noir - Paul Cezanne; Potrait - Lucian Freud). From left to right of each row, the effect of stroke is increasingly amplified.

Fig. 16.

The styled image with stroke basis intervened using spectrum based methods. The left most row shows the style images (From top to bottom: A Muse (La Muse) - Pablo Picasso; Number 4 (Gray and Red) - Jackson Pollock; Shipwreck - J.M.W. Turner; Natura Morta - Giorgio Morandi; The Scream - Edvard Munch). From left to right of each row, the effect of stroke is increasingly amplified. (Color figure online)

Fig. 17.

The styled image with stroke basis intervened using ICA. The left most row shows the style images (From top to bottom: The Great Wave off Kanagawa - Katsushika Hokusai; Composition - Alberto Magnelli; Dancer - Ernst Ludwig Kirchner; Pistachio Tree in the Courtyard of the Chateau Noir - Paul Cezanne; Potrait - Lucian Freud). From left to right of each row, the effect of stroke is increasingly amplified.

Fig. 18.

The styled image with stroke basis intervened using ICA. The left most row shows the style images (From top to bottom: A Muse (La Muse) - Pablo Picasso; Number 4 (Gray and Red) - Jackson Pollock; Shipwreck - J.M.W. Turner; Natura Morta - Giorgio Morandi; The Scream - Edvard Munch). From left to right of each row, the effect of stroke is increasingly amplified. (Color figure online)

Appendix 3: Stroke Intervention

We demonstrate more styled images with stroke basis intervened using spectrum based method (Figs. 15 and 16) and ICA (Figs. 17 and 18) respectively.

Appendix 4: Style Mixing

We demonstrate more styled images transferred with compound style generated by mixing the color basis and stroke basis of two different styles. The results of both spectrum based method and ICA method are shown in Fig. 19 with comparison with traditional mixing method - interpolation.

Fig. 19.

The left two columns are the style images used for mixing. Specifically, we mix the color of the most left one with the stroke of the second left one. The third left column shows the styled images with traditional interpolation method. The second right column shows the styled images using spectrum mixing method. The most right column shows the styled images using ICA mixing method.