Conditional Image Repainting via Semantic Bridge and Piecewise Value Function

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12354)


We study conditional image repainting where a model is trained to generate visual content conditioned on user inputs, and composite the generated content seamlessly onto a user provided image while preserving the semantics of users’ inputs. The content generation community has been pursuing to lower the skill barriers. The usage of human language is the rose among thorns for this purpose, because the language is friendly to users but poses great difficulties for the model in associating relevant words with the semantically ambiguous regions. To resolve this issue, we propose a delicate mechanism which bridges the semantic chasm between the language input and the generated visual content. The state-of-the-art image compositing techniques pose a latent ceiling of fidelity for the composited content during the adversarial training process. In this work, we improve the compositing by breaking through the latent ceiling using a novel piecewise value function. We demonstrate on two datasets that the proposed techniques can better assist tackling conditional image repainting compared to the existing ones.


Image generation Semantic Compositing Adversarial 



PKU affiliated authors are supported by National Natural Science Foundation of China under Grant No. 61872012, National Key R&D Program of China (2019YFF0302902), and Beijing Academy of Artificial Intelligence (BAAI).

Supplementary material

504446_1_En_27_MOESM1_ESM.pdf (5.9 mb)
Supplementary material 1 (pdf 6059 KB)


  1. 1.
  2. 2.
    Caesar, H., Uijlings, J.R.R., Ferrari, V.: Coco-stuff: thing and stuff classes in context. In: CVPR (2018)Google Scholar
  3. 3.
    Chen, B., Kae, A.: Toward realistic image compositing with adversarial learning. In: CVPR (2019)Google Scholar
  4. 4.
    Chen, X., Qing, L., He, X., Luo, X., Xu, Y.: FTGAN: a fully-trained generative adversarial networks for text to face generation. CoRR abs/1904.05729 (2019)Google Scholar
  5. 5.
    Cong, W., et al.: Deep image harmonization via domain verification. CoRR abs/1911.13239 (2019)Google Scholar
  6. 6.
    Cun, X., Pun, C.: Improving the harmony of the composite image by spatial-separated attention module. CoRR abs/1907.06406 (2019)Google Scholar
  7. 7.
    Goodfellow, I.J., et al.: Generative adversarial nets. In: NIPS (2014)Google Scholar
  8. 8.
    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a nash equilibrium. In: NIPS (2017)Google Scholar
  9. 9.
    Huang, H., Xu, S., Cai, J., Liu, W., Hu, S.: Temporally coherent video harmonization using adversarial networks. TIP 29, 214–224 (2020)MathSciNetGoogle Scholar
  10. 10.
    Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)Google Scholar
  11. 11.
    Li, W., et al.: Object-driven text-to-image synthesis via adversarial training. In: CVPR (2019)Google Scholar
  12. 12.
    Li, Y., Singh, K.K., Ojha, U., Lee, Y.J.: Mixnmatch: multifactor disentanglement and encoding for conditional image generation. CoRR abs/1911.11758 (2019)Google Scholar
  13. 13.
    Park, T., Liu, M., Wang, T., Zhu, J.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR (2019)Google Scholar
  14. 14.
    Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP (2014)Google Scholar
  15. 15.
    Qiao, T., Zhang, J., Xu, D., Tao, D.: Mirrorgan: learning text-to-image generation by redescription. In: CVPR (2019)Google Scholar
  16. 16.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  17. 17.
    Tan, H., Liu, X., Li, X., Zhang, Y., Yin, B.: Semantics-enhanced adversarial nets for text-to-image synthesis. In: ICCV (2019)Google Scholar
  18. 18.
    Tripathi, S., Chandra, S., Agrawal, A., Tyagi, A., Rehg, J.M., Chari, V.: Learning to generate synthetic data via compositing. In: CVPR (2019)Google Scholar
  19. 19.
    Tsai, Y., Shen, X., Lin, Z., Sunkavalli, K., Lu, X., Yang, M.: Deep image harmonization. In: CVPR (2017)Google Scholar
  20. 20.
    Tsai, Y., Shen, X., Lin, Z., Sunkavalli, K., Yang, M.: Sky is not the limit: semantic-aware sky replacement. TOG 35, 149 (2016)CrossRefGoogle Scholar
  21. 21.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Technical report CNS-TR-2011-001, California Institute of Technology (2011)Google Scholar
  22. 22.
    Wang, T., Liu, M., Zhu, J., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional gans. In: CVPR (2018)Google Scholar
  23. 23.
    Weng, S., Li, W., Li, D., Jin, H., Shi, B.: Misc: multi-condition injection and spatially-adaptive compositing for conditional person image synthesis. In: CVPR (2020)Google Scholar
  24. 24.
    Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: towards realistic high-resolution image blending. In: ACM MM (2019)Google Scholar
  25. 25.
    Xu, T., et al.: Attngan: fine-grained text to image generation with attentional generative adversarial networks. In: CVPR (2018)Google Scholar
  26. 26.
    Yin, G., Liu, B., Sheng, L., Yu, N., Wang, X., Shao, J.: Semantics disentangling for text-to-image generation. In: CVPR (2019)Google Scholar
  27. 27.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: ICCV (2019)Google Scholar
  28. 28.
    Zhou, P., Han, X., Morariu, V.I., Davis, L.S.: Learning rich features for image manipulation detection. In: CVPR (2018)Google Scholar
  29. 29.
    Zhu, M., Pan, P., Chen, W., Yang, Y.: DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis. In: CVPR (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.NELVT, Department of Computer Science and TechnologyPeking UniversityBeijingChina
  2. 2.Samsung Research America AI CenterMountain ViewUSA
  3. 3.Institute for Artificial IntelligencePeking UniversityBeijingChina

Personalised recommendations