Skip to main content
SpringerLink
Log in

Feature fusion and decomposition: exploring a new way for Chinese calligraphy style classification

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Chinese calligraphy is an invaluable legacy of Chinese culture, since it bears great artistic and aesthetic value. In this paper, we aim at the problem of Chinese calligraphy style classification, which is an important branch of Chinese calligraphy study. Chinese calligraphy style classification remains a challenging task due to its dramatic intra-class difference and tiny inter-class difference. Therefore, we propose a novel CNN embedded with feature fusion and feature decomposition modules to solve this problem. We first fuse the features of several images from the same category to augment their potential style-related features. Then we feed the fused feature to an attention module to decompose it to two components, viz. style-related feature and style-unrelated feature. We further apply two types of loss function to jointly supervise our network. On the one hand, we feed the style-related feature to a style classifier which is supervised by cross-entropy loss. On the other hand, we construct a correlation loss based on the Pearson correlation coefficient to make the two decomposed features as orthogonal as possible. By optimizing these two types of loss simultaneously, our proposed network has obtained the accuracies of \(98.63\%\) and \(94.35\%\) respectively on two datasets. Besides, substantial experiments demonstrate the effectiveness of the feature fusion and decomposition modules. The proposed approach compares favorably with state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Huang, C., Xue, H.: The China Academic Digital Associative Library (CADAL). In: Digital libraries and institutional repositories: breakthroughs in research and practice, pp 67–77 (2020)

  2. Gao, P., Wu, J., Lin, Y., Xia, Y., Mao, T.: Fast Chinese calligraphic character recognition with large-scale data. Multimed. Tools Appl. 74(17), 7221–7238 (2015)

    Article  Google Scholar 

  3. Li, M., Wang, J., Yang, Y., Huang, W., Du, W.: Improving GAN-based calligraphy character generation using graph matching. In: 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion, Sofia, Bulgaria, pp. 291–295 (2019)

  4. Jiang, H., Yang, G., Huang, K., Zhang, R.: W-net: One-shot arbitrary-style Chinese character generation with deep neural networks. In: International Conference on Neural Information Processing, Siem Reap, Cambodia, pp. 483–493 (2018)

  5. Zhou, P., Zhao, Z., Zhang, K., Li, C., Wang, C.: An end-to-end model for Chinese calligraphy generation. Multimed. Tools Appl. 80(5), 6737–6754 (2021)

    Article  Google Scholar 

  6. Bi, F., Han, J., Tian, Y., Wang, Y.: SSGAN: generative adversarial networks for the stroke segmentation of calligraphic characters. Vis. Comput. 38(7), 2581–2590 (2022)

    Article  Google Scholar 

  7. Xiang, L., Zhao, Y., Dai, G., Gou, R., Zhang, H., Shi, J.: The study of Chinese calligraphy font style based on edge-guided filter and convolutional neural network. In: 2020 IEEE 5th International Conference on Signal and Image Processing, Nanjing, China, pp. 883–887 (2020)

  8. Dai, F., Tang, C., Lv, J.: Classification of calligraphy style based on convolutional neural network. In: International Conference on Neural Information Processing, Siem Reap, Cambodia, pp. 359–370 (2018)

  9. Zhang, J., Guo, M., Fan, J.: A novel CNN structure for fine-grained classification of Chinese calligraphy styles. Int. J. Doc. Anal. Recogn. 22(2), 177–188 (2019)

    Article  Google Scholar 

  10. Liu, L., Cheng, W., Qiu, T., Tao, C., Chen, Q., Lu, Y., Suen, C. Y.: Multi-loss siamese convolutional neural network for Chinese calligraphy style classification. In: International Conference on Neural Information Processing, Bali, Indonesia, pp. 425–432 (2021)

  11. Zhang, X., Nagy, G.: Style comparisons in calligraphy. In: Document Recognition and Retrieval XIX, vol. 8297, p. 82970 (2012)

  12. Mu, Y., Liu, X., Wang, L.: A Pearson’s correlation coefficient based decision tree and its parallel implementation. Inf. Sci. 435, 40–58 (2018)

    Article  MathSciNet  Google Scholar 

  13. Slimane, F., Kanoun, S., Hennebert, J., Alimi, A.M., Ingold, R.: A study on font-family and font-size recognition applied to Arabic word images at ultra-low resolution. Pattern Recogn. Lett. 34(2), 209–218 (2013)

    Article  ADS  Google Scholar 

  14. Zhang, S., Jin, L., Tao, D., Yang, Z.: A faster method for Chinese font recognition based on Harris corner. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics, Manchester, UK, pp. 4271–4275 (2013)

  15. Ma, J., Jiang, X., Fan, A., Jiang, J., Yan, J.: Image matching from handcrafted to deep features: a survey. Int. J. Comput. Vision 129(1), 23–79 (2021)

    Article  MathSciNet  Google Scholar 

  16. Tao, D., Jin, L., Zhang, S., Yang, Z., Wang, Y.: Sparse discriminative information preservation for Chinese character font categorization. Neurocomputing 129, 159–167 (2014)

    Article  Google Scholar 

  17. Song, W., Lian, Z., Tang, Y., Xiao, J.: Content-independent font recognition on a single Chinese character using sparse representation. In: 2015 13th International Conference on Document Analysis and Recognition, Tunis, Tunisia, pp. 376–380 (2015)

  18. Zhang, Y., Liu, Y., He, J., Zhang, J.: Recognition of calligraphy style based on global feature descriptor. In: 2013 IEEE International Conference on Multimedia and Expo, San Jose, Canada, pp. 1–6 (2013)

  19. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, pp. 770–778 (2016)

  20. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K. Q.: Densely connected convolutional networks. In: the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, pp. 4700–4708 (2017)

  21. Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In: the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 6848–6856 (2018)

  22. Tan, M., Le, Q.: EfficientNet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, Long Beach, CA, USA, pp. 6105–6114 (2019)

  23. Yang, H., Fan, Y., Lv, G., Liu, S., Guo, Z.: Exploiting emotional concepts for image emotion recognition. The Visual Computer, pp. 1–14 (2022)

  24. Lv, G., Dong, L., Zhang, W., Xu, W.: Region-based adaptive association learning for robust image scene recognition. Visual Comput., pp. 1–21 (2022)

  25. Phaphuangwittayakul, A., Ying, F., Guo, Y., Zhou, L., Chakpitak, N.: Few-shot image generation based on contrastive meta-learning generative adversarial network. Visual Comput., pp. 1–14 (2022)

  26. Zhang, Y., Han, S., Zhang, Z., Wang, J., Bi, H.: CF-GAN: cross-domain feature fusion generative adversarial network for text-to-image synthesis. Visual Comput., pp. 1–11 (2022)

  27. Kera, S. B., Tadepalli, A., Ranjani, J. J.: A paced multi-stage block-wise approach for object detection in thermal images. Visual Comput., pp. 1–17 (2022)

  28. Zhao, H., Yang, D., Yu, J.: 3D target detection using dual domain attention and sift operator in indoor scenes. Visual Comput., pp. 1–10 (2021)

  29. Tao, D., Lin, X., Jin, L., Li, X.: Principal component 2-D long short-term memory for font recognition on single Chinese characters. IEEE Trans. Cybern. 46(3), 756–765 (2015)

    Article  PubMed  Google Scholar 

  30. Wang, Y., Lian, Z., Tang, Y., Xiao, J.: Font recognition in natural images via transfer learning. In: International Conference on Multimedia Modeling, Bangkok, Thailand, pp. 229–240 (2018)

  31. Huang, S., Zhong, Z., Jin, L., Zhang, S., Wang, H.: DropRegion training of inception font network for high-performance Chinese font recognition. Pattern Recogn. 77, 395–411 (2018)

    Article  ADS  Google Scholar 

  32. Guo, M., Xu, T., Liu, J., Liu, Z., Jiang, P., Mu, T., Zhang, S., Martin, R. R., Cheng, M., Hu, S.: Attention mechanisms in computer vision: a survey. Comput. Visual Media, pp. 1–38 (2022)

  33. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 7132–7141 (2018)

  34. Qin, Z., Zhang, P., Wu, F., Li, X.: Fcanet: Frequency channel attention networks. In: the IEEE/CVF International Conference on Computer Vision, Montreal, Canada, pp. 783–792 (2021)

  35. Zhang, H., Zu, K., Lu, J., Zou, Y., Meng, D.: EPSANet: an efficient pyramid squeeze attention block on convolutional neural network. arXiv preprint arXiv:2105.14447 (2021)

  36. Hu, J., Shen, L., Albanie, S., Sun, G., Vedaldi, A.: Gather-excite: exploiting feature context in convolutional neural networks. Adv. Neural Inf. Process. Syst. 31 (2018)

  37. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., Shlens, J.: Stand-alone self-attention in vision models. Adv. Neural Inf. Process. Syst. 32 (2019)

  38. Woo, S., Park, J., Lee, J., Kweon, I. S.: CBAM: Convolutional block attention module. In: European Conference on Computer Vision, Munich, Germany, pp. 3–19 (2018)

  39. Park, J., Woo, S., Lee, J., Kweon, I. S.: BAM: Bottleneck attention module. In: British Machine Vision Conference, Newcastle, UK (2018)

  40. Zhang, J., Yu, W., Wang, Z., Li, J., Pan, Z.: Attention-enhanced CNN for chinese calligraphy styles classification. In: 2021 IEEE 7th International Conference on Virtual Reality, Foshan, China, pp. 352–358 (2021)

  41. Chen, J., Mu, S., Xu, S., Ding, Y.: HENet: Forcing a network to think more for font recognition. In: 2021 3rd International Conference on Advanced Information Science and System, Sanya, China, pp. 1–5 (2021)

  42. Wang, Y., Gong, D., Zhou, Z., Ji, X., Wang, H., Li, Z., Liu, W., Zhang, T.: Orthogonal deep features decomposition for age-invariant face recognition. In: European Conference on Computer Vision, Munich, Germany, pp. 738–753 (2018)

  43. Meng, L., Yan, C., Li, J., Yin, J., Liu, W., Xie, H., Li, L.: Multi-features fusion and decomposition for age-invariant face recognition. In: the 28th ACM International Conference on Multimedia, Seattle, US, pp. 3146–3154 (2020)

  44. Huang, Z., Zhang, J., Shan, H.: When age-invariant face recognition meets face age synthesis: A multi-task learning framework. In: the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, US, pp. 7282–7291 (2021)

  45. Liu, L., Lao, S., Fieguth, P.W., Guo, Y., Wang, X., Pietikäinen, M.: Median robust extended local binary pattern for texture classification. IEEE Trans. Image Process. 25(3), 1368–1381 (2016)

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  46. Omid-Zohoor, A., Young, C., Ta, D., Murmann, B.: Toward always-on mobile object detection: energy versus performance tradeoffs for embedded HOG feature extraction. IEEE Trans. Circuits Syst. Video Technol. 28(5), 1102–1115 (2017)

    Article  Google Scholar 

  47. Yu, C., Zhao, X., Zheng, Q., Zhang, P., You, X.: Hierarchical bilinear pooling for fine-grained visual recognition. In: European Conference on Computer Vision, Munich, Germany, pp. 574–589 (2018)

  48. Tan, M., Wang, G., Zhou, J., Peng, Z., Zheng, M.: Fine-grained classification via hierarchical bilinear pooling with aggregated slack mask. IEEE Access 7, 117944–117953 (2019)

    Article  Google Scholar 

  49. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

  50. Pervaiz, N., Fraz, M., Shahzad, M.: Per-former: rethinking person re-identification using transformer augmented with self-attention and contextual mapping. Visual Comput., pp. 1–16 (2022)

  51. Yan, F., Silamu, W., Li, Y., Chai, Y.: SPCA-net: a based on spatial position relationship co-attention network for visual question answering. Visual Comput., pp. 1–12 (2022)

  52. Zhao, T., Pan, S., Gao, W., Sheng, C., Sun, Y., Wei, J.: Attention Unet++ for lightweight depth estimation from sparse depth samples and a single RGB image. Vis. Comput. 38(5), 1619–1630 (2022)

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by National Natural Science Foundation of China under Grant 61603256 and the Natural Sciences and Engineering Research Council of Canada.

Author information

Authors and Affiliations

  1. School of Mathematics and Computer Sciences, Nanchang University, Nanchang, 330031, China

    Yong Zhou, Li Liu & Taorong Qiu

  2. Xi’an Microelectronics Technology Institute, Xi’an, 710065, China

    Hui Ma

  3. School of Communication and Electronic Engineering, East China Normal University, Shanghai, 200241, China

    Yue Lu

  4. Centre for Pattern Recognition and Machine Intelligence, Concordia University, Montreal, H3G 1M8, Canada

    Ching Y. Suen

Authors
  1. Yong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Taorong Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yue Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ching Y. Suen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Ma, H., Liu, L. et al. Feature fusion and decomposition: exploring a new way for Chinese calligraphy style classification. Vis Comput 40, 1631–1642 (2024). https://doi.org/10.1007/s00371-023-02875-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02875-1

Keywords

Search

Navigation