Abstract
Semantic image synthesis, a pivotal task in image-to-image translation, has been widely addressed using generative adversarial network (GAN) models. However, existing GAN-based approaches often suffer from inadequate incorporation of structural and spatial information, resulting in unsatisfactory quality of the synthesized images and a pronounced disparity between photo-realistic and generated images. In this paper, we propose a novel GAN-based methodology to address these limitations, enabling the generation of high-resolution images from semantic label maps while bridging the quality gap and preserving detailed information in the generated outputs. The proposed approach leverages a two-step process, starting with a local binary pattern convolutional generator that produces a local binary pattern feature map. Subsequently, a global convolutional generator is fed with the segmentation map and the feature map through a learned modulation scheme facilitated by a multi-feature adaptive denormalization layer (MFADE) during the training process to generate photo-realistic images. Extensive experiments using Cityscapes, ADE20K, and COCO-stuff datasets validate the performance of our proposed method and showcase its accuracy and robustness in addressing semantic image synthesis tasks, thereby paving the way for its potential applications in enhancing urban sensing and data analytics in Smart Cities. The source code is available at https://github.com/karimmagdy/ULBPGAN.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kingma, D.P., Welling, M.: Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114 (2013)
Kingma, D.P., Mohamed, S., Rezende, D.J., Welling, M.: Semi-supervised learning with deep generative models. In: Advances in Neural Information Processing Systems, pp. 3581–3589 (2014)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems, pp. 4790–4798 (2016)
Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: Artificial Intelligence and Statistics, pp. 448–455 (2009 )
Bengio, Y., Laufer, E., Alain, G., Yosinski, J.: Deep generative stochastic networks trainable by backprop. In: International Conference on Machine Learning, pp. 226–234 (2014)
Karim Magdy, G.K., Abbas, H.: Semantic image synthesis manipulation for stability problem using generative adversarial networks A survey. Comput. Inf. Bull. (2021)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1125–1134 (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Wang, T.-C., Liu, M.-Y., Zhu,J.-Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8798–8807 (2018)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2009, pp. 248–255. IEEE (2009)
Park, T., Liu, M.-Y., Wang, T.-C., Zhu, J.-Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2337–2346 (2019)
Liu, X., Yin, G., Shao, J., Wang, X., et al.: Learning to predict layout-to-image conditional convolutions for semantic image synthesis. In: Advances in Neural Information Processing Systems, pp. 570–580 (2019)
Sushko, V., Schönfeld, E., Zhang, D., Gall, J., Schiele, B., Khoreva, A.: You only need adversarial supervision for semantic image synthesis. arXiv preprint arXiv:2012.04781 (2020)
Tan, Z., et al.: Efficient semantic image synthesis via class-adaptive normalization. IEEE Trans. Pattern Anal. Mach. Intell. 44, 4852–4866 (2021)
Wang, Y., Qi, L., Chen, Y.-C., Zhang, X., Jia, J.: Image synthesis via semantic composition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13 749–13 758 (2021)
Tan, Z., et al.: Diverse semantic image synthesis via probability distribution modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7962–7971 (2021)
Shi, Y., Liu, X., Wei, Y., Wu, Z., Zuo, W.: Retrieval-based spatially adaptive normalization for semantic image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11 224–11 233 (2022)
Wang, W., et al.: Semantic image synthesis via diffusion models. arXiv preprint arXiv:2207.00050 (2022)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
Ulyanov, D., Vedaldi, A., Lempitsky, A.: Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
Krizhevsky, A., Sutskever, I. Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Zhu, P., Abdal, R., Qin, Y., Wonka, P.: Sean: image synthesis with semantic region-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5104–5113 (2020)
Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recogn. 29(1), 51–59 (1996)
Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3021, pp. 469–481. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24670-1_36
Jia, S., Deng, B., Zhu, J., Jia, X., Li, Q.: Local binary pattern-based hyperspectral image classification with superpixel guidance. IEEE Trans. Geosci. Remote Sens. 56(2), 749–759 (2017)
Wu, H., Zhou, J.: Privacy leakage of sift features via deep generative model based image reconstruction. arXiv preprint arXiv:2009.01030 (2020)
Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)
Dhingra, S., Bansal, P.: Experimental analogy of different texture feature extraction techniques in image retrieval systems. Multimedia Tools Appl 79(37), 27391–27406 (2020)
Wei, X., Yu, X., Liu, B., Zhi, L.: Convolutional neural networks and local binary patterns for hyperspectral image classification. Eur. J. Remote Sens. 52(1), 448–462 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633–641 (2017)
Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209–1218 (2018)
Acknowledgments
We acknowledge BA-HPC, The Bibliotheca Alexandrina High-Performance Computing, for their facility that offers us merit-based access to a High-Performance Computing cluster to conduct our experiments.
Funding
“This research received no external funding”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Conflicts of Interest
Declare conflicts of interest or state “The authors declare no conflict of interest.”
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Magdy, K., Khoriba, G., Abbas, H. (2024). Enhancing Semantic Image Synthesis: A GAN-Based Approach with Multi-Feature Adaptive Denormalization Layer. In: Mosbah, M., Kechadi, T., Bellatreche, L., Gargouri, F. (eds) Model and Data Engineering. MEDI 2023. Lecture Notes in Computer Science, vol 14396. Springer, Cham. https://doi.org/10.1007/978-3-031-49333-1_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-49333-1_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49332-4
Online ISBN: 978-3-031-49333-1
eBook Packages: Computer ScienceComputer Science (R0)