Style-Neutralized Pattern Classification Based on Adversarially Trained Upgraded U-Net

  • Haochuan Jiang
  • Kaizhu HuangEmail author
  • Rui Zhang
  • Amir Hussain


Traditional machine learning approaches usually hold the assumption that data for model training and in real applications are created following the identical and independent distribution (i.i.d.). However, several relevant research topics have demonstrated that such condition may not always describe the real scenarios. One particular case is that the patterns are equipped with diverse and changeable style information. In this paper, a novel classification framework named Style Neutralization Generative Adversarial Classifier (SN-GAC), based on an upgraded U-Net architecture, and trained adversarially with the Generative Adversarial Network (GAN) framework, is introduced to accomplish the classification in such disparate and inconsistent data information case. The generative model in SN-GAC neutralizes style information from the original style-discriminative patterns (style-source) by building the mapping function from them to their style-free counterparts (corresponding standard examples, standard-target). A well-learned generator in the SN-GAC framework is capable of producing the targeted style-neutralized data (generated-target), satisfying the i.i.d. condition. Additionally, SN-GAC is trained adversarially, where an independent discriminator is used to surveil and supervise the training progress of the above-mentioned generator by distinguishing between the real and the generated. Simultaneously, an auxiliary classifier is also embedded in the discriminator to assign the correct class label of both the real and generated data. This process proves effective to aid the generator to produce high-quality human-readable style-neutralized patterns. It will then be further fine-tuned for the sake of promoting the final classification performance. Extensive experiments have adequately demonstrated the effectiveness of the proposed SN-GAC framework: it outperforms several relevant state-of-the-art baselines on two empirical data sets in the non-i.i.d. data classification task.


Style neutralization Generative adversarial network Pattern classification 



Acknowledgment goes to Ms. Zijun CUI who offered assistance in designing several of the illustrations in this paper.

Funding Information

The work reported here was partially supported by the following: National Natural Science Foundation of China under grant no. 61876155; Natural Science Fund for Colleges and Universities in Jiangsu Province under grant no. 17KJD520010; Suzhou Science and Technology Program under grant no. SYG2-01712, SZS201613; Jiangsu University Natural Science Research Programme under grant no. 17KJB-520041; Key Program Special Fund in XJTLU (KSF-A-01).

Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants performed by any of the authors.


  1. 1.
    Jiang H, Huang K, Zhang R. Field support vector regression. Proceedings of the International Conference on Neural Information Processing. Cham: Springer; 2017.Google Scholar
  2. 2.
    Huang K, Jiang H, Zhang X. Field support vector machines. Proceedings of the 1st International Conference on Internet of Things and Machine Learning. ACM; 2017.Google Scholar
  3. 3.
    Huang K, Jiang H, Zhang X. Field support vector machines. IEEE Transactions on Emerging Topics in Computational Intelligence 2017;1.6:454–63.CrossRefGoogle Scholar
  4. 4.
    Zhang X-Y, Huang K, Liu C-L. Pattern field classification with style normalized transformation. Twenty-Second International Joint Conference on Artificial Intelligence; 2011.Google Scholar
  5. 5.
    Liu Z-Y, Qiao H, Yang X, Hoi SCH. Graph matching by simplified convex-concave relaxation procedure. Int J Comput Vis. 2014;109(3).Google Scholar
  6. 6.
    Liu Z-Y, Qiao H. GNCCP-Graduated nonconvexityand concavity procedure. IEEE Trans Pattern Anal Mach Intell. 2013;36(6).Google Scholar
  7. 7.
    Gourier N, Hall D, Crowley JL. Estimating face orientation from robust detection of salient facial features. ICPR International Workshop on Visual Observation of Deictic Gestures; 2004.Google Scholar
  8. 8.
    Liu C-L, Yin F, Wang D-H, Wang Q-F. CASIA online and offline Chinese handwriting databases. 2011 International Conference on Document Analysis and Recognition. IEEE; 2011.Google Scholar
  9. 9.
    Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems.Google Scholar
  10. 10.
    Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017.Google Scholar
  11. 11.
    Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention. Cham: Springer; 2015.Google Scholar
  12. 12.
    Lvmin Z, Ji Y, Lin X, Liu C. Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier gan. 4th IAPR Asian Conference on Pattern Recognition. IEEE; 2017.Google Scholar
  13. 13.
    Odena A, Olah C, Shlens J. Conditional image synthesis with auxiliary classifier gans. Proceedings of the 34th International Conference on Machine Learning; 2017.Google Scholar
  14. 14.
    Mirza M, Osindero S. Conditional generative adversarial nets. arXiv:1411.1784.
  15. 15.
    Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A, Chen X. Improved techniques for training gans. Advances in Neural Information Processing Systems. 2016.Google Scholar
  16. 16.
    Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC. Improved training of wasserstein gans. Advances in Neural Information Processing Systems. 2017.Google Scholar
  17. 17.
    Yoshida Y, Miyato T. 2017. Spectral norm regularization for improving the generalizability of deep learning. arXiv:1705.10941.
  18. 18.
    Miyato T, Kataoka T, Koyama M, Yoshida Y. Spectral normalization for generative adversarial networks. arXiv:1802.05957. 2018.
  19. 19.
    Antoniou A, Storkey A, Edwards H. Data augmentation generative adversarial networks. arXiv:1711.04340. 2017.
  20. 20.
    Shrivastava A, Pfister T, Tuzel O, Susskind J, Wang W, Webb R. Learning from simulated and unsupervised images through adversarial training. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017.Google Scholar
  21. 21.
    Wang T-C, Liu M-Y, Zhu J-Y, Tao A, Kautz J, Catanzaro B. High-resolution image synthesis and semantic manipulation with conditional gans. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018.Google Scholar
  22. 22.
    Taigman Y, Polyak A, Wolf L. Unsupervised cross-domain image generation. arXiv:1611.02200. 2016.
  23. 23.
    Zhu J-Y, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision; 2017.Google Scholar
  24. 24.
    Jiang H, Yang G, Huang K, Zhang R. W-Net: one-shot arbitrary-style chinese character generation with deep neural networks. Proceedings of the International Conference on Neural Information Processing; 2018.Google Scholar
  25. 25.
    Evgeniou T, Pontil M. Regularized multi-task learning. Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM; 2004.Google Scholar
  26. 26.
    Sarkar P, Nagy G. Style consistent classification of isogenous patterns. IEEE Trans Pattern Anal Mach Intell 2005;27:1.CrossRefGoogle Scholar
  27. 27.
    Tenenbaum JB, Freeman WT. Separating style and content with bilinear models. Neural Comput 2000;12: 6.CrossRefGoogle Scholar
  28. 28.
    Zhong G, Huang K. 2018. Semi-supervised learning: background, applications and future directions. Nova Science Publishers Inc.Google Scholar
  29. 29.
    Hsu C-W, Lin C-J. A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 2002;2:13.Google Scholar
  30. 30.
    He K, et al. Identity mappings in deep residual networks. European conference on computer vision. Cham: Springer; 2016.Google Scholar
  31. 31.
    Huang K, Hussain A, Wang Q, Zhang R. Deep learning: fundamentals, theory, and applications. Springer; 2019. ISBN-13: 978-3540794516.Google Scholar
  32. 32.
    Jiang Y, Lian Z, Tang Y, Xiao J. 2017. DCFOnt: an end-to-end deep chinese font generation system. SIGGRAPH Asia 2017 Technical Briefs. ACM.Google Scholar
  33. 33.
    Johnson M, Schuster M, Le QV, Krikun M, Wu Y, Chen Z, Thorat N, et al. 2017. Google’s multilingual neural machine translation system: enabling zero-shot translation. Transactions of the Association for Computational Linguistics 5.Google Scholar
  34. 34.
  35. 35.
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 2014;1:15.Google Scholar
  36. 36.
    Cortes C, Vapnik V. Support-vector networks. Mach Learn 1995;20:3.Google Scholar
  37. 37.
    Cate H, Dalvi F, Hussain Z. Deepface: face generation using deep learning. Proceedings of the IEEE conference on computer vision and pattern recognition; 2014.Google Scholar
  38. 38.
    Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems; 2012.Google Scholar
  39. 39.
    Jing X-Y, Wong H-S, Zhang D. Face recognition based on 2D Fisherface approach. Pattern Recogn 2006; 4:39.Google Scholar
  40. 40.
    Kimura F, Takashina K, Tsuruoka S, Miyake Y. 1987. Modified quadratic discriminant functions and the application to Chinese character recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence 1.Google Scholar
  41. 41.
    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, et al. Tensorflow: a system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16); 2016.Google Scholar
  42. 42.
    Wall ME, Rechtsteiner A, Rocha LM. Singular value decomposition and principal component analysis. A practical approach to microarray data analysis. Boston: Springer; 2003.Google Scholar
  43. 43.
    Boser BE, Guyon IM, Vapnik VN. A training algorithm for optimal margin classifiers. Proceedings of the Fifth Annual Workshop on Computational Learning Theory. ACM; 1992.Google Scholar
  44. 44.
    Internet Archive. GB 2312-1980: information technology—Chinese ideogram coded character set for information interchange (basic set).

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of EEEXi’an Jiaotong - Liverpool University, SIPSuzhouPeople’s Republic of China
  2. 2.Department of MSXi’an Jiaotong - Liverpool University, SIPSuzhouPeople’s Republic of China
  3. 3.Cyber and Cognitive Big Data LabEdinburgh Napier UniversityScotlandUK

Personalised recommendations