Continuous and Diverse Image-to-Image Translation via Signed Attribute Vectors

Abstract

Recent image-to-image (I2I) translation algorithms focus on learning the mapping from a source to a target domain. However, the continuous translation problem that synthesizes intermediate results between two domains has not been well-studied in the literature. Generating a smooth sequence of intermediate results bridges the gap of two different domains, facilitating the morphing effect across domains. Existing I2I approaches are limited to either intra-domain or deterministic inter-domain continuous translation. In this work, we present an effectively signed attribute vector, which enables continuous translation on diverse mapping paths across various domains. In particular, we introduce a unified attribute space shared by all domains that utilize the sign operation to encode the domain information, thereby allowing the interpolation on attribute vectors of different domains. To enhance the visual quality of continuous translation results, we generate a trajectory between two sign-symmetrical attribute vectors and leverage the domain information of the interpolated results along the trajectory for adversarial training. We evaluate the proposed method on a wide range of I2I translation tasks. Both qualitative and quantitative results demonstrate that the proposed framework generates more high-quality continuous translation results against the state-of-the-art methods.

Appendices

Appendix A Additional Experiments

1.1 A.1 More Continuous Translation Results

We present more diverse continuous translation paths from the source domain to the target one in this section. We can continuously translate an input image $I_s$ in the source domain to multiple images $I_{t_1}, I_{t_2}, \dots , I_{t_N}$ in the target domain. We name this type as one input and various targets. The target attribute vector can be obtained from either extracting a reference image sampled from the target domain (reference-guided) or randomly generating an SAV of the target domain (latent-guided).

Reference-Guided Continuous Translation Figures 13, 14, 15, and 16 present more reference-guided continuous translation in both style translation and shape-variation tasks.

Latent-Guided Continuous Translation Figures 17 and 18 show more latent-guided continuous translation results in both style translation and shape-variation tasks, which uses randomly sampled signed attribute vectors of target domains.

Fig. 13

Reference-guided continuous translation results on the CelebA-HQ dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 14

Reference-guided continuous translation on the AFHQ dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 15

Reference-guided continuous translation results on the Yosemite dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 16

Reference-guided continuous translation on the Photo2Artwork dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 17

Latent-guided continuous translation on the CelebA-HQ and AFHQ dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 18

Latent-guided continuous translation on the Photo2Artwork dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 19

One input, one target, and diverse intermediate points. The source image can be translated into one specific target with various intermediate points. We show three translation paths on cat $\rightarrow$ wildlife task with different intermediate points

Fig. 20

Reference-guided continuous expression translation on the CelebA-HQ dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 21

Latent-guided continuous expression translation on the CelebA-HQ dataset. Green and blue bounding boxes denote generated images of the source and target domain, respectively (Color figure online)

Fig. 22

Translation from female $\rightarrow$ male images. a Translation based on reference images (top); Translation based on interpolated domain labels or randomly sampled latent vectors (bottom). Green and blue bounding boxes denote generated images of the source and target domain, respectively. b From left to right, the y-axis of each sub-figure are target domain translation ACC (the larger, the better), target domain FID (the smaller, the better), DIPD (the smaller, the better), and LPIPS between two adjacent interpolated images (the smaller, the better). Each curve is plotted under different $\beta$ values. Solid lines indicate methods using reference images. Dash lines denote approaches using interpolated domain labels or randomly sampled latent vectors of the target domain (Color figure online)

Fig. 23

Translation from cat $\rightarrow$ dog images. a Translation based on reference images (top); translation based on interpolated domain labels or randomly sampled latent vectors (bottom). Green and blue bounding boxes denote generated images of the source and target domain, respectively. b From left to right, the y-axis of each sub-figure are target domain translation ACC (the larger, the better), target domain FID (the smaller, the better), DIPD (the smaller, the better), and LPIPS between two adjacent interpolated images (the smaller, the better). Each curve is plotted under different $\beta$ values. Solid lines indicate methods using reference images. Dash lines denote approaches using interpolated domain labels or randomly sampled latent vectors of the target domain (Color figure online)

Fig. 24

Additional ablation studies on the male $\rightarrow$ female translation. a Green and blue bounding boxes denote generated images of the source and target domain, respectively. b The x-axis is the $\beta$. From left to the right, the y-axis in each sub-figure are target domain translation ACC (the larger, the better), target domain FID (the smaller, the better), DIPD (the smaller, the better), and LPIPS between two adjacent interpolated images (the smaller, the better), respectively (Color figure online)

1.2 A.2 Multiple Translation Paths using Diverse Intermediate Points

Given a source image $I_s$ and a target image $I_t$, we can obtain multiple continuous translation paths by passing different intermediate attribute vectors, called one input, one target, and diverse intermediate points. In particular, we can apply interpolation with multiple intermediate points between a source attribute vector and a target attribute vector to generate multiple translation paths, as presented in Fig. 19.

1.3 A.3 More Results on Facial Expression Continuous Translation

We present more results of facial expression continuous translation on the CelebA-HQ dataset. Figures 20 and 21 show the reference-guided and latent-guided continuous translation results, respectively.

1.4 A.4 More Comparisons with State-of-the-arts

We present more quantitative and qualitative comparisons results in Figs. 22 and 23.

1.5 A.5 More Ablation Studies

We conduct additional ablation studies on other loss objectives commonly used for image-to-image translation in Fig. 24.

Cycle-Consistency Loss Both quantitative and qualitative results demonstrate that ${\mathcal {L}}_{1}^{\mathrm {cc}}$ helps preserve the consistency of domain-invariant characteristics of generated images. Without ${\mathcal {L}}_{1}^{\mathrm {cc}}$, the model achieves the highest DIPD value when $\beta = 1.0$, as illustrated in the “DIPD vs. $\beta$” curve of Fig. 24b. Figure 24a also shows that the expression of generated image changes from the neutral into the smile when $\beta >0.4$.

Self-Reconstruction Loss The model trained without ${\mathcal {L}}_{1}^{\mathrm {recon}}$ cannot reconstruct the original image well, as shown in Fig. 24a at $\beta = 0$. Therefore, the “DIPD vs. $\beta$” curve of Fig. 24b demonstrates that it obtains the highest DIPD value when $\beta = 0$.

Fig. 25

Comparisons of the content representations of two domains on CelebA-HQ dataset using t-SNE. Each data point is a content representation encoded from an image of that domain

Content Adversarial Loss The “FID vs. $\beta$” curve of Fig. 24b shows that the full model has better FID values than the one without ${\mathcal {L}}_{\mathrm {adv}}^{\mathrm {content}}$. We also observe that the full model aligns content representations of two domains better than that trained without ${\mathcal {L}}_{\mathrm {adv}}^{\mathrm {content}}$, as shown in Fig. 25. Thus, the content discriminator helps align the distribution of the content representations of two domains and further disentangles the content and attribute representations.

Latent Regression and Mode Seeking Constraints To better illustrate the effectiveness of ${\mathcal {L}}_{1}^{\mathrm {latent}}$ and ${\mathcal {L}}_{\mathrm {ms}}$, we calculate the LPIPS score of generated images on the male $\rightarrow$ female translation for diversity comparisons. As shown in Table 3, both ${\mathcal {L}}_{1}^{\mathrm {latent}}$ and ${\mathcal {L}}_\mathrm {{ms}}$ facilitate improving the diversity of generated images. In addition, we also observe that the model trained without ${\mathcal {L}}_\mathrm {{ms}}$ cannot capture the style of the reference image for the translated image, as shown in Fig. 24a. The “FID vs. $\beta$” curve of Fig. 24b further indicates that training with ${\mathcal {L}}_{\mathrm {ms}}$ enhances the quality of generated images.

Table 3 Diversity comparisons on the male $\rightarrow$ female translation

Appendix B: Network Architecture

Table 4 shows the network configuration details where Conv(k, s, p) and DeConv(k, s, p) denote the convolutional layer and transposed convolutional layer with k as kernel size, s as stride, and p as padding; DownResBlock and UpResBlock adopt the average pooling for down-sampling and the nearest-neighbor interpolation for up-sampling respectively; LN is the layer normalization (Ba et al. 2016) and IN is the instance normalization (Ulyanov et al. 2016); AdaIN is the adaptive instance normalization (Huang and Belongie 2017); and LReLU indicates leaky ReLU (Maas et al. 2013) with a negative slope of 0.2.

Table 4 Details of network architectures: (a) content encoder $E_c$ architecture, (b) generator G architecture. Style translation tasks directly concatenate the attribute vector, while shape-variation translation tasks adopt AdaIN to inject the attribute vector, (c) attribute encoder $E_a$ and discriminator D architecture. The d is set to 8 and 1 for $E_a$ and D respectively, (d) content discriminator $D_c$ architecture. The c is set to 256 and 512 for style translation, and shape-variation translation tasks, respectively, and shape-variation translation tasks do not contain the first DownResBlock, (e) fusing network F architecture

Appendix C: The Video

We present translation videos at https://helenmao.github.io/SAVI2I/. The video shows some examples of generating diverse animations from a source input image. In particular, this is accomplished by continuously translating the input image to the target domains. We present reference-guided continuous translation and latent-guided continuous translation of the proposed method.