Abstract
In multiclass deep network classifiers, the burden of classifying samples of different classes is put on a single classifier. As the result, the optimum classification accuracy is not obtained. Also, training times are large due to running the CNN training on single CPU/GPU. However, it is known that using ensembles of classifiers increases the performance. Also, the training times can be reduced by running each member of the ensemble on a separate processor. Ensemble learning has been used in the past for traditional methods to a varying extent and is a hot topic. With the advent of deep learning, ensemble learning has been applied to the former as well. However, an area which is unexplored and has potential is one-versus-all (OVA) deep ensemble learning. In this paper, we explore it and show that by using OVA ensembles of deep networks, improvements in performance of deep networks can be obtained. As shown in this paper, the classification capability of deep networks can be further increased by using an ensemble of binary classification (OVA) deep networks. We implement a novel technique for the case of digit image recognition and test and evaluate it on the same. In the proposed approach, a single OVA deep network classifier is dedicated to each category. Subsequently, OVA deep network ensembles have been investigated. Every network in an ensemble has been trained by an OVA training technique using the stochastic gradient descent with momentum algorithm (SGDMA). For classification of a test sample, the sample is presented to each network in the ensemble. After prediction score voting, the network with the largest score is assumed to have classified the sample. The experimentation has been done on the MNIST digit dataset, the USPS + digit dataset, and MATLAB digit image dataset. Our proposed technique outperforms the baseline on digit image recognition for all datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
X. Dong, Z. Yu, W. Cao, Y. Shi, Q. Ma, A survey on ensemble learning. Frontiers Comput. Sci. 14(2), 241–258 (2020). https://doi.org/10.1007/s11704-019-8208-z
C. Kandaswamy, L.M. Silva, L.A. Alexandre, J.M. Santos, Deep transfer learning ensemble for classification, in Advances in Computational Intelligence (Springer International Publishing, Cham, 2015), pp. 335–348
D. Nozza, E. Fersini, E. Messina, Deep learning and ensemble methods for domain adaptation, in 2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI), 6–8 November 2016, pp. 184-189. https://doi.org/10.1109/ictai.2016.0037
X. Liu, Z. Liu, G. Wang, Z. Cai, H. Zhang, Ensemble transfer learning algorithm. IEEE Access 6, 2389–2396 (2018). https://doi.org/10.1109/ACCESS.2017.2782884
Y. Freund, R. Schapire, N. Abe, A short introduction to boosting. J. Jpn. Soc. Artif. Intell. 14(771–780), 1612 (1999)
E. Dikici, L.M. Prevedello, M. Bigelow, R.D. White, B.S. Erdal, Constrained generative adversarial network ensembles for sharable synthetic data generation. arXiv preprint arXiv:200300086 (2020)
Z. Yu, Y. Zhang, C.L.P. Chen, J. You, H. Wong, D. Dai, S. Wu, J. Zhang, Multiobjective semisupervised classifier ensemble. IEEE Trans. Cybern. 49(6), 2280–2293 (2019). https://doi.org/10.1109/TCYB.2018.2824299
Z. Yu, Y. Zhang, J. You, C.L.P. Chen, H. Wong, G. Han, J. Zhang, Adaptive semi-supervised classifier ensemble for high dimensional data classification. IEEE Trans. Cybern. 49(2), 366–379 (2019). https://doi.org/10.1109/TCYB.2017.2761908
H.I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, P. Muller, Deep neural network ensembles for time series classification, in 2019 International Joint Conference on Neural Networks (IJCNN), 14–19 July 2019, pp. 1-6. https://doi.org/10.1109/ijcnn.2019.8852316
S. Tao, Deep neural network ensembles, in Machine Learning, Optimization, and Data Science (Springer International Publishing, Cham, 2019), pp. 1–12
S. Sun, S. Wang, Y. Wei, G. Zhang, A clustering-based nonlinear ensemble approach for exchange rates forecasting. IEEE Trans. Syst. Man Cybern. Syst. (2018)
O. Sagi, L. Rokach, Ensemble learning: a survey. WIREs Data Min. Knowl. Discov. 8(4), (2018). https://doi.org/10.1002/widm.1249
K. Yang, Z. Yu, X. Wen, W. Cao, C.L.P. Chen, H. Wong, J. You, Hybrid classifier ensemble for imbalanced data. IEEE Trans. Neural Netw. Learn. Syst. 31(4), 1387–1400 (2020). https://doi.org/10.1109/TNNLS.2019.2920246
J. Zheng, X. Cao, B. Zhang, X. Zhen, X. Su, Deep ensemble machine for video classification. IEEE Trans. Neural Netw. Learn. Syst. 30(2), 553–565 (2019). https://doi.org/10.1109/TNNLS.2018.2844464
A.M. Hafiz, G.M. Bhat, A survey of deep learning techniques for medical diagnosis, in Information and Communication Technology for Sustainable Development (Springer, 2020), pp. 161–170
A. Madakannu, A. Selvaraj, DIGI-Net: a deep convolutional neural network for multi-format digit recognition. Neural Comput. Appl. 32(15), 11373–11383 (2020). https://doi.org/10.1007/s00521-019-04632-9
D. Mellouli, T.M. Hamdani, J.J. Sanchez-Medina, M.B. Ayed, A.M. Alimi, Morphological convolutional neural network architecture for digit recognition. IEEE Trans. Neural Netw. Learn. Syst. 30(9), 2876–2885 (2019). https://doi.org/10.1109/TNNLS.2018.2890334
S. Ali, Z. Shaukat, M. Azeem, Z. Sakhawat, T. Mahmood, K. ur Rehman, An efficient and improved scheme for handwritten digit recognition based on convolutional neural network. SN Appl. Sci. 1(9), 1125 (2019). https://doi.org/10.1007/s42452-019-1161-5
J. Qiao, G. Wang, W. Li, M. Chen, An adaptive deep Q-learning strategy for handwritten digit recognition. Neural Netw. 107, 61–71 (2018). https://doi.org/10.1016/j.neunet.2018.02.010
M. Mozafari, M. Ganjtabesh, A. Nowzari-Dalini, S.J. Thorpe, T. Masquelier, Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks. Pattern Recogn. 94, 87–95 (2019). https://doi.org/10.1016/j.patcog.2019.05.015
S.R. Kulkarni, B. Rajendran, Spiking neural networks for handwritten digit recognition—supervised learning and network optimization. Neural Netw. 103, 118–127 (2018). https://doi.org/10.1016/j.neunet.2018.03.019
X.-X. Niu, C.Y. Suen, A novel hybrid CNN–SVM classifier for recognizing handwritten digits. Pattern Recogn. 45(4), 1318–1325 (2012). https://doi.org/10.1016/j.patcog.2011.09.021
A. Bellili, M. Gilloux, P. Gallinari, An MLP-SVM combination architecture for offline handwritten digit recognition. Int. J. Doc. Anal. Recogn. 5(4), 244–252 (2003). https://doi.org/10.1007/s10032-002-0094-4
MNIST Dataset, http://yann.lecun.com/exdb/mnist/
USPS Dataset, https://cs.nyu.edu/~roweis/data.html
Y. Chen, E. Keogh, B. Hu, N. Begum, A. Bagnall, A. Mueen, G. Batista, The UCR time series classification archive (2015)
S. Xie, R. Girshick, P. Dollár, Z. Tu, K. He, Aggregated residual transformations for deep neural networks, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 21–26 July 2017, pp. 5987-5995. https://doi.org/10.1109/cvpr.2017.634
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), pp. 770–778
K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks, in European Conference on Computer Vision (Springer, 2016), pp. 630–645
A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images (2009)
X. Wei, H. Yu, Y. Hu, Y. Zhang, R. Weng, W. Luo, Multiscale collaborative deep models for neural machine translation. arXiv preprint arXiv:200414021 (2020)
A.M. Hafiz, G.M. Bhat, A survey on instance segmentation: state of the art. Int. J. Multimedia Inform. Retrieval 9(3), 171–189 (2020). https://doi.org/10.1007/s13735-020-00195-x
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Hafiz, A.M., Hassaballah, M. (2021). Digit Image Recognition Using an Ensemble of One-Versus-All Deep Network Classifiers. In: Kaiser, M.S., Xie, J., Rathore, V.S. (eds) Information and Communication Technology for Competitive Strategies (ICTCS 2020). Lecture Notes in Networks and Systems, vol 190. Springer, Singapore. https://doi.org/10.1007/978-981-16-0882-7_38
Download citation
DOI: https://doi.org/10.1007/978-981-16-0882-7_38
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-0881-0
Online ISBN: 978-981-16-0882-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)