Skip to main content
SpringerLink
Log in

An unsupervised font style transfer model based on generative adversarial networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Chinese characters, because of their complex structure and a large number, lead to an extremely high cost of time for designers to design a complete set of characters. As a result, the dramatic growth of characters used in various fields such as culture and business has formed a strong contradiction between supply and demand with Chinese font design. Although most of the existing Chinese characters transformation models greatly alleviate the demand for character usage, the semantics of the generated characters cannot be guaranteed and the generation efficiency is low. At the same time, the models require large amounts of paired data for training, which requires a large amount of sample processing time. To address the problems of existing methods, this paper proposes an unsupervised Chinese characters generation method based on generative adversarial networks, which fuses Style-Attentional Net to a skip-connected U-Net as a GAN generator network architecture. It effectively and flexibly integrates local style patterns based on the semantic spatial distribution of content images while retaining feature information of different sizes. Our model generates fonts that maintain the source domain content features and the target domain style features at the end of training. The addition of the style specification module and the classification discriminator allows the model to generate multiple style typefaces. The generation results show that the model proposed in this paper can perform the task of Chinese character style transfer well. The model generates high-quality images of Chinese characters and generates Chinese characters with complete structures and natural strokes. In the quantitative comparison experiments and qualitative comparison experiments, our model has more superior visual effects and image performance indexes compared with the existing models. In sample size experiments, clearly structured fonts are still generated and the model demonstrates significant robustness. At the same time, the training conditions of our model are easy to meet and facilitate generalization to real applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Atarsaikhan G, Iwana BK, Narusawa A, Yanai K, Uchida S (2017, November). Neural font style transfer. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR) (Vol. 5, pp. 51-56). IEEE.

  2. Azadi, S., Fisher, M., Kim, V. G., Wang, Z., Shechtman, E., & Darrell, T. (2018). Multi-content Gan for few-shot font style transfer. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7564-7573).

  3. Baxter W, Govindaraju N (2010, February). Simple data-driven modeling of brushes. In proceedings of the 2010 ACM SIGGRAPH symposium on interactive 3D graphics and games (pp. 135-142).

  4. Chang B, Zhang Q, Pan S, Meng L (2018, March) Generating handwritten chinese characters using cyclegan. In 2018 IEEE winter conference on applications of computer vision (WACV) (pp. 199-207). IEEE.

  5. Chen J, Ji Y, Chen H, Xu X (2019) Learning one-to-many stylised Chinese character transformation and generation by generative adversarial networks. IET Image Process 13(14):2680–2686

    Article  Google Scholar 

  6. Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8789–8797).

  7. Gao, Y., & Wu, J. (2020, April). GAN-based unpaired Chinese character image translation via skeleton transformation and stroke rendering. In proceedings of the AAAI conference on artificial intelligence (Vol. 34, no. 01, pp. 646-653).

  8. Gholizadeh-Ansari M, Alirezaie J, Babyn P (2020) Deep learning for low-dose CT denoising using perceptual loss and edge detection layer. J Digit Imaging 33(2):504–515

    Article  Google Scholar 

  9. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y (2014) Generative adversarial networks. arXiv preprint arXiv:1406.2661.

  10. Hayashi H, Abe K, Uchida S (2019) GlyphGAN: style-consistent font generation based on generative adversarial networks. Knowl-Based Syst 186:104927

    Article  Google Scholar 

  11. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium Advances in neural information processing systems:30

  12. Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708).

  13. Huang, Y., He, M., Jin, L., & Wang, Y. (2020, August). RD-GAN: few/zero-shot Chinese characters style transfer via radical decomposition and rendering. In European conference on computer vision (pp. 156-172). Springer, Cham.49

  14. Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125-1134).

  15. Ji G, Wang Z, Zhou L, Xia Y, Zhong S, Gong S (2020) SAR image colorization using multidomain cycle-consistency generative adversarial network. IEEE Geosci Remote Sens Lett 18(2):296–300

    Article  Google Scholar 

  16. Jiang, Y., Lian, Z., Tang, Y., & Xiao, J. (2017). Dcfont: an end-to-end deep chinese font generation system. In SIGGRAPH Asia 2017 technical briefs (pp. 1-4).

  17. Jiang, Y., Lian, Z., Tang, Y., & Xiao, J. (2019, July). Scfont: structure-guided chinese font generation via deep stacked networks. In proceedings of the AAAI conference on artificial intelligence (Vol. 33, no. 01, pp. 4015-4022).

  18. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

  19. Lake BM, Salakhutdinov R, Tenenbaum JB (2015) Human-level concept learning through probabilistic program induction. Science 350(6266):1332–1338

    Article  MathSciNet  Google Scholar 

  20. Lee J (1999) Simulating oriental black-ink painting. IEEE Comput Graph Appl 19(3):74–81

    Article  Google Scholar 

  21. Lian, Z., Zhao, B., & Xiao, J. (2016). Automatic generation of large-scale handwriting fonts via style learning. In SIGGRAPH Asia 2016 technical briefs (pp. 1-4).

  22. Lian Z, Zhao B, Chen X, Xiao J (2018) EasyFont: a style learning-based system to easily build your large-scale handwriting fonts. ACM Transactions on Graphics (TOG) 38(1):1–18

    Article  Google Scholar 

  23. Minoofam SAH, Dehshibi MM, Bastanfard A, Eftekhari P (2012) Ad-hoc Ma'qeli script generation using block cellular automata. J Cell Autom 7(4):321–334

    MATH  Google Scholar 

  24. Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784.

  25. Modhej N, Bastanfard A, Teshnehlab M, Raiesdana S (2020) Pattern separation network based on the hippocampus activity for handwritten recognition. IEEE Access 8:212803–212817

    Article  Google Scholar 

  26. Musto, L., & Zinelli, A. (2020). Semantically adaptive image-to-image translation for domain adaptation of semantic segmentation. arXiv preprint arXiv:2009.01166.

  27. Park, D. Y., & Lee, K. H. (2019). Arbitrary style transfer with style-attentional networks. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5880-5888).

  28. Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.

  29. Ronneberger O, Fischer P & Brox, T (2015, October) U-net: convolutional networks for biomedical image segmentation. In international conference on medical image computing and computer-assisted intervention (pp. 234-241). Springer. Cham.

  30. Sun, D., Ren, T., Li, C., Su, H., & Zhu, J. (2017). Learning to write stylized chinese characters by reading a handful of examples. arXiv preprint arXiv:1712.06424.

  31. Tian Y (2020) :ReWrite.https://github.com/kaonashityc/Rewrite/, accessed August 2020

  32. Velek, O., Liu, C. L., & Nakagawa, M. (2001, September). Generating realistic kanji character images from on-line patterns. In proceedings of sixth international conference on document analysis and recognition (pp. 556-560). IEEE.

  33. Wang, Y., Wang, H., Pan, C., & Fang, L. (2008, March). Style preserving Chinese character synthesis based on hierarchical representation of character. In 2008 IEEE international conference on acoustics, speech and signal processing (pp. 1097-1100). IEEE.

  34. Wang D, Li C, Song H, Xiong H, Liu C, He D (2020) Deep learning approach for apple edge detection to remotely monitor apple growth in orchards. IEEE Access 8:26911–26925

    Article  Google Scholar 

  35. Wen, C., Pan, Y., Chang, J., Zhang, Y., Chen, S., Wang, Y., ... & Tian Q (2021) Handwritten Chinese font generation with collaborative stroke refinement. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 3882–3891).

  36. Wong HT, Ip HH (2000) Virtual brush: a model-based synthesis of Chinese calligraphy. Comput Graph 24(1):99–113

    Article  Google Scholar 

  37. Wu, Y., Zhuang, Y., Pan, Y., & Wu, J. (2006, July). Web based chinese calligraphy learning with 3-d visualization method. In 2006 IEEE international conference on multimedia and expo (pp. 2073-2076). IEEE.

  38. Wu, L., Chen, X., Meng, L., & Meng, X. (2020, July). Multitask adversarial learning for Chinese font style transfer. In 2020 international joint conference on neural networks (IJCNN) (pp. 1-8). IEEE.

  39. Xu S, Lau FC, Cheung WK, Pan Y (2005) Automatic generation of artistic Chinese calligraphy. IEEE Intell Syst 20(3):32–39

    Article  Google Scholar 

  40. Yamini K, Swetha KS, Prasanna PL, Swathi MRV, Maddumala VR (2020) Image colorization with deep convolutional open cv. Journal of Engineering Science 11(4):533–543

    Google Scholar 

  41. Yang S, Wang J, Deng B, Liu C, Li H, Fietkiewicz C, Loparo KA (2018) Real-time neuromorphic system for large-scale conductance-based spiking neural networks. IEEE transactions on cybernetics 49(7):2490–2503

    Article  Google Scholar 

  42. Yang S, Deng B, Wang J, Li H, Lu M, Che & Loparo, K. A. (2019) Scalable digital neuromorphic architecture for large-scale biophysically meaningful neural network with multi-compartment neurons. IEEE transactions on neural networks and learning systems 31(1):148–162

    Article  Google Scholar 

  43. Yang S, Gao T, Wang J, Deng B, Lansdell B, Linares-Barranco B (2021) Efficient spike-driven learning with dendritic event-based processing. Front Neurosci 15:97

    Google Scholar 

  44. Yang S, Wang J, Hao X, Li H, Wei X, Deng B, Loparo KA (2021) BiCoSS: toward large-scale cognition brain with multigranular neuromorphic architecture. IEEE Transactions on Neural Networks and Learning Systems.

    Google Scholar 

  45. Yang, S., Wang, J., Zhang, N., Deng, B., Pang, Y., & Azghadi, M. R. (2021). CerebelluMorphic: large-scale neuromorphic model and architecture for supervised motor learning. IEEE Transactions on Neural Networks and Learning Systems.

    Google Scholar 

  46. Yuchen T (2017) “Zi2zi: master chinese calligraphy with conditional adversarial networks” https://github.com/kaonashi-tyc/zi2zi. Accessed 27 Nov 2020

  47. Zhang, Z., Wu, J., & Yu, K. (2010, June). Chinese calligraphy specific style rendering system. In proceedings of the 10th annual joint conference on digital libraries (pp. 99-108).

  48. Zhang XY, Yin F, Zhang YM, Liu CL, Bengio Y (2017) Drawing and recognizing chinese characters with recurrent neural network. IEEE Trans Pattern Anal Mach Intell 40(4):849–862

    Article  Google Scholar 

  49. Zhang, Y., Zhang, Y., & Cai, W. (2018). Separating style and content for generalized style transfer. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8447-8455).

  50. Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2019, May). Self-attention generative adversarial networks. In international conference on machine learning (pp. 7354-7363). PMLR.

  51. Zhou, B., Wang, W., & Chen, Z. (2011, July). Easy generation of personal Chinese handwritten fonts. In 2011 IEEE international conference on multimedia and expo (pp. 1-6). IEEE.

  52. Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In proceedings of the IEEE international conference on computer vision (pp. 2223-2232).

  53. Zhu, X., Hu, H., Lin, S., & Dai, J. (2019). Deformable convnets v2: more deformable, better results. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9308-9316).

Download references

Author information

Authors and Affiliations

  1. Physics and Telecommunication Engineering, South China Normal University, Guangzhou, China

    Sihan Zeng & Zhongliang Pan

Authors
  1. Sihan Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongliang Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhongliang Pan.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, S., Pan, Z. An unsupervised font style transfer model based on generative adversarial networks. Multimed Tools Appl 81, 5305–5324 (2022). https://doi.org/10.1007/s11042-021-11777-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11777-0

Keywords

Search

Navigation