Improving discrimination ability of convolutional neural networks by hybrid learning

Original Paper
First Online:
Received:
Revised:
Accepted:

Abstract

The discrimination of similar patterns is important because they are the major sources of the classification error. This paper proposes a novel method to improve the discrimination ability of convolutional neural networks (CNNs) by hybrid learning. The proposed method embeds a collection of discriminators as well as a recognizer in a shared CNN. By visualizing contrastive class saliency, we show that learning with embedded discriminators leads the shared CNN to detect and catch the differences among similar classes. Also proposed is a hybrid learning algorithm that learns recognition and discrimination together. The proposed method learns recognition focusing on the differences among similar classes, and thereby improves the discrimination ability of the CNN. Unlike conventional discrimination methods, the proposed method does not require predefined sets of similar classes or additional step to integrate its result with that of the recognizer. In experiments on two handwritten Hangul databases SERI95a and PE92, the proposed method reduced classification error from 2.56 to 2.33, and from 4.04 to 3.66 % respectively. These improvement lead to relative error reduction rates of 8.97 % on SERI95a, and 9.42 % on PE92. Our best results update the state-of-the-art performance which were 4.04 % on SERI95a and 7.08 % on PE92.

Keywords

Deep learning Convolutional neural networks Hybrid learning Discrimination Character recognition Machine learning Pattern recognition 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

This work was supported by the DGIST R&D Program of the Ministry of Science, ICT and Future Planning (15-IT-03).

References

  1. 1.
    Kim, I.-J., Kim, J.: Pairwise discrimination based on a stroke importance measure. Pattern Recognit. 35(10), 2259–2266 (2002)CrossRefMATHGoogle Scholar
  2. 2.
    Leung, K.C., Leung, C.H.: Recognition of handwritten Chinese characters by critical region analysis. Pattern Recognit. 43(3), 949–961 (2010)CrossRefMATHGoogle Scholar
  3. 3.
    Xu, B., Huang, K., Liu, C.L.: Similar characters recognition by critical region selection based on average symmetric uncertainty. In: Proceedings of 12th ICFHR, Kolkata, India, pp. 527–532 (2010)Google Scholar
  4. 4.
    Ryu, S.-J., Kim, I.-J.: Discrimination of similar characters using nonlinear normalization based on regional importance measure. Int. J. Doc. Anal. Recognit. (IJDAR) 17(1), 79–89 (2014)CrossRefGoogle Scholar
  5. 5.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  6. 6.
    Kim, I.-J., Xie, X.: Handwritten Hangul recognition using deep convolutional neural networks. Int. J. Doc. Anal. Recognit. 18(1), 1–13 (2014)CrossRefGoogle Scholar
  7. 7.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556 (2014)
  8. 8.
    Yin, F., Wang, Q.F., Zhang, X.X.Y., Liu, C.L.: ICDAR 2013 Chinese handwriting recognition competition. http://www.nlpr.ia.ac.cn/events/CHRcompetition2013/competition/ICDAR%202013%20CHR%20competition.pdf
  9. 9.
    Kim, D.-I., Kim, S.-Y., Lee, S.-W.: Design and construction of a large-set off-line handwritten hangul character image database KU-1. In: Proceedings of \(9{{\rm th}}\) National Conference on Korean Language Information Processing, pp. 152–159 (1997) (in Korean)Google Scholar
  10. 10.
    Kim, D.H., Bang, S.Y.: An overview of hangul handwritten image database PE92. In: Proceedings of 4\({{\rm th}}\) National Conference on Korean Language Information Processing, pp. 152–159 (1992) (in Korean)Google Scholar
  11. 11.
    Kim, I.-J., Kim, J.: Pairwise discrimination based on a stroke importance measure. Pattern Recognit. 35(10), 2259–2266 (2002)Google Scholar
  12. 12.
    Goodfellow, I. et al.: Maxout networks, arXiv:1302.4389 (2013)
  13. 13.
    Lin, M., Chen, Q., Yan, S.: Network in network. In: Proceedings of ICLR (2014)Google Scholar
  14. 14.
    Szegedy, C. et al.: Going deeper with convolutions, arXiv:1409.4842 (2014)
  15. 15.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv:1502.03167 (2015)
  16. 16.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network, arXiv:1503.02531 (2015)
  17. 17.
    Girshick, R. et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE (2014)Google Scholar
  18. 18.
    Socher, R. et al.: Convolutional-recursive deep learning for 3d object classification. Advances in Neural Information Processing Systems (2012)Google Scholar
  19. 19.
    Zeiler, M., Fergus, R.: Visualizing and understanding convolutional networks. Compu Vis-ECCV 2014, 818–833 (2014)Google Scholar
  20. 20.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps, arXiv:1312.6034 (2013)
  21. 21.
    Sato, A., Yamada, K.: Generalized learning vector quantization. Adv. Neural Inf. Process. Syst. 15(8), 423–429 (1996)Google Scholar
  22. 22.
    Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Bengio, Y. et al.: Curriculum learning. In: Proceedings of 26th Annual International Conference on Machine Learning. ACM, pp. 41–48 (2009)Google Scholar
  24. 24.
    Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis, null. IEEE, p. 958 (2003)Google Scholar
  25. 25.
    Srivastava, N., et al.: Dropout: a simple way to prevent neural networks from overfitting. J Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.School of CSEEHandong Global UniversityPohangKorea
  2. 2.School of Creative Convergence EducationHandong Global UniversityPohangKorea
  3. 3.Department of IoT and Robotics Convergence ResearchDGISTDaeguKorea

Personalised recommendations