Skip to main content
SpringerLink
Log in

A deep contractive autoencoder for solving multiclass classification problems

  • Research Paper
  • Published:
Evolutionary Intelligence Aims and scope Submit manuscript

Abstract

Contractive auto encoder (CAE) is on of the most robust variant of standard Auto Encoder (AE). The major drawback associated with the conventional CAE is its higher reconstruction error during encoding and decoding process of input features to the network. This drawback in the operational procedure of CAE leads to its incapability of going into finer details present in the input features by missing the information worth consideration. Resultantly, the features extracted by CAE lack the true representation of all the input features and the classifier fails in solving classification problems efficiently. In this work, an improved variant of CAE is proposed based on layered architecture following feed forward mechanism named as deep CAE. In the proposed architecture, the normal CAEs are arranged in layers and inside each layer, the process of encoding and decoding take place. The features obtained from the previous CAE are given as inputs to the next CAE. Each CAE in all layers are responsible for reducing the reconstruction error thus resulting in obtaining the informative features. The feature set obtained from the last CAE is given as input to the softmax classifier for classification. The performance and efficiency of the proposed model has been tested on five MNIST variant-datasets. The results have been compared with standard SAE, DAE, RBM, SCAE, ScatNet and PCANet in term of training error, testing error and execution time. The results revealed that the proposed model outperform the aforementioned models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Aamir M, Mohd Nawi N, Mahdin HB, Naseem R, Zulqarnain M (2020) Auto-encoder variants for solving handwritten digits classification problem. Int J Fuzzy Logic Intell Syst 20(1):8–16

    Article  Google Scholar 

  2. Hosseini-Asl E, Zurada JM, Nasraoui O (2016) Deep learning of part-based representation of data using sparse autoencoders with nonnegativity constraints. IEEE Trans Neural Netw Learn Syst 27(12):2486–2498

    Article  Google Scholar 

  3. Bengio Y, Yao L, Alain G, Vincent P (2013) Generalized denoising auto-encoders as generative models. In: Advances in neural information processing systems pp 899–907

  4. Jia K, Sun L, Gao S, Song Z, Shi BE (2015) Laplacian auto-encoders: an explicit learning of nonlinear data manifold. Neurocomputing 160:250–260

    Article  Google Scholar 

  5. Zhang N, Ding S, Shi Z (2016) Denoising Laplacian multi-layer extreme learning machine. Neurocomputing 171:1066–1074

    Article  Google Scholar 

  6. Wang W, Cui Z, Chang H, Shan S, Chen X (2014) Deeply coupled auto-encoder networks for cross-view classification. arXiv preprint arXiv:14022031

  7. Liu W, Ma T, Tao D, You J (2016) HSAE: a Hessian regularized sparse auto-encoders. Neurocomputing 187:59–65

    Article  Google Scholar 

  8. Chorowski J, Zurada JM (2015) Learning understandable neural networks with nonnegative weight constraints. IEEE Trans Neural Netw Learn Syst 26(1):62–69

    Article  MathSciNet  Google Scholar 

  9. Hong C, Yu J, Wan J, Tao D, Wang M (2015) Multimodal deep autoencoder for human pose recovery. IEEE Trans Image Process 24(12):5659–5670

    Article  MathSciNet  Google Scholar 

  10. Nishino K, Inaba M (2016) Bayesian AutoEncoder: generation of Bayesian networks with hidden nodes for features. In: AAAI pp 4244–4245

  11. Rifai S, Mesnil G, Vincent P, Muller X, Bengio Y, Dauphin Y, et al. (2011) Higher order contractive auto-encoder. In: Joint European conference on machine learning and knowledge discovery in databases. Springer. pp 645–660

  12. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11:3371–3408

    MathSciNet  MATH  Google Scholar 

  13. Qi Y, Wang Y, Zheng X, Wu Z (2014) Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. pp 6716–6720

  14. Lemme A, Reinhart RF, Steil JJ (2010) Efficient online learning of a non-negative sparse autoencoder. In: ESANN. Citeseer

  15. Wu P, Hoi SC, Xia H, Zhao P, Wang D, Miao C (2013) Online multimodal deep similarity learning with application to image retrieval. In: Proceedings of the 21st ACM international conference on multimedia. ACM, pp 153–162

  16. Mescheder L, Nowozin S, Geiger A (2017) Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. arXiv preprint arXiv:1701.04722

  17. Liu Y, Feng X, Zhou Z (2016) Multimodal video classification with stacked contractive autoencoders. Sig Process 120:761–766

    Article  Google Scholar 

  18. Rifai S, Vincent P, Muller X, Glorot X, Bengio Y (2011) Contractive auto-encoders: explicit invariance during feature extraction. In: Proceedings of the 28th international conference on international conference on machine learning. Omnipress, pp 833–840

  19. Nawi NM, Ransing RS, Salleh MNM, Ghazali R, Hamid NA (2010) An improved back propagation neural network algorithm on classification problems. In: Database theory and application, bio-science and bio-technology. Springer, pp 177–188

  20. Wahid F, Ismail LH, Ghazali R, Aamir M (2019) An efficient artificial intelligence hybrid approach for energy management in intelligent buildings. KSII Trans Internet Inf Syst 13(12):5904–5927

    Google Scholar 

  21. Mohd Nawi N, Hamzah F, Hamid NA, Rehman MZ, Aamir M, Azhar AR (2017) An optimized back propagation learning algorithm with adaptive learning rate. Int J Adv Sci Eng Inf Technol 7(5):1693–1700

    Article  Google Scholar 

  22. Snášel V, Nowaková J, Xhafa F, Barolli L (2017) Geometrical and topological approaches to Big Data. Future Gener Comput Syst 67:286–296

    Article  Google Scholar 

  23. Wahid F, Ghazali R (2018) Hybrid of firefly algorithm and pattern search for solving optimization problems. Evol Intel 1–10

  24. Izenman AJ (2013) Linear discriminant analysis. In: Modern multivariate statistical techniques. Springer, pp 237–280

  25. Wahid F, Ghazali R, Fayaz M, Shah AS (2017) Statistical features based approach (SFBA) for hourly energy consumption prediction using neural network. Networks 8:9

    Google Scholar 

  26. Aamir M, Mohd Nawi N, Wahid F, Mahdin H (2019) An efficient normalized restricted Boltzmann machine for solving multiclassclassification problems. Int J Adv Comput Sci Appl 10(8):416–426

    Google Scholar 

  27. Wahid F, Ghazali R, Shah H (2018) An improved hybrid firefly algorithm for solving optimization problems. In: International conference on soft computing and data mining. Springer, pp 14–23

  28. Larochelle H, Erhan D, Courville A, Bergstra J, Bengio Y (2007) An empirical evaluation of deep architectures on problems with many factors of variation. In: Proceedings of the 24th international conference on Machine learning. ACM, pp 473–480

  29. Chollet F, et al (2018) Keras: the python deep learning library. Astrophysics Source Code Library

  30. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al (2016) Tensorflow: a system for large-scale machine learning. In: OSDI. vol 16, pp 265–283

  31. Montavon G, Müller KR, Cuturi M (2016) Wasserstein training of restricted Boltzmann machines. In: Advances in neural information processing systems, pp 3718–3726

  32. Liang J, Liu R (2015) Stacked denoising autoencoder and dropout together to prevent overfitting in deep neural network. In: 2015 8th International congress on image and signal processing (CISP). IEEE, pp 697–701

  33. Sankaran A, Vatsa M, Singh R, Majumdar A (2017) Group sparse autoencoder. Image Vis Comput 60:64–74

    Article  Google Scholar 

  34. Chan TH, Jia K, Gao S, Lu J, Zeng Z, Ma Y (2015) PCANet: A simple deep learning baseline for image classification? IEEE Trans Image Process 24(12):5017–5032

    Article  MathSciNet  Google Scholar 

  35. Zi Y, Xie F, Jiang Z (2018) A cloud detection method for Landsat 8 images based on PCANet. Remote Sens 10(6):877

    Article  Google Scholar 

  36. Zhu W, Miao J, Qing L, Huang GB (2015) Hierarchical extreme learning machine for unsupervised representation learning. In: 2015 International joint conference on neural networks (IJCNN). IEEE, pp 1–8

  37. Liu R, Lu T (2016) Character recognition based on PCANet. In: 2016 15th International symposium on parallel and distributed computing (ISPDC). IEEE, pp 364–367

  38. Lee JN, Byeon YH, Pan SB, Kwak KC (2018) An EigenECG network approach based on PCANet for personal identification from ECG Signal. Sensors. https://doi.org/10.3390/s18114024

    Article  Google Scholar 

  39. Wang Z, Chang S, Ling Q, Huang S, Hu X, Shi H, et al (2016) Stacked approximated regression machine: a simple deep learning approach. arXiv preprint arXiv:1608.04062.

  40. Soon FC, Khaw HY, Chuah JH, Kanesan J (2018) PCANet-Based Convolutional Neural Network Architecture For a Vehicle Model Recognition System. IEEE Trans Intell Transp Syst 99:1–11

    Google Scholar 

  41. Liao L, Jin W, Pavel R (2016) Enhanced restricted Boltzmann machine with prognosability regularization for prognostics and health assessment. IEEE Trans Industr Electron 63(11):7076–7083

    Article  Google Scholar 

  42. Biglari F, Ebadian A (2015) Limited memory BFGS method based on a high-order tensor model. Comput Optim Appl 60(2):413–422

    Article  MathSciNet  Google Scholar 

  43. Jae-Neung Lee YHB, Kwak KC (2018) An EigenECG network approach based on PCANet for personal identification from ECG signal. Micromachines 9(4):411

    Google Scholar 

  44. Lee JN, Byeon YH, Kwak KC (2018) Design of ensemble stacked auto-encoder for classification of horse gaits with MEMS inertial sensor technology. Micromachines 9(8):411

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Universiti Tun Hussein Onn Malaysia (UTHM) and Ministry of Higher Education (MOHE) Malaysia for supporting this Research.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn, Batu Pahat, Malaysia

    Muhammad Aamir & Hairulnizam Mahdin

  2. Soft Computing and Data Mining Center, Universiti Tun Hussein Onn, Batu Pahat, Malaysia

    Nazri Mohd Nawi

  3. Faculty of Computer Science and Information Technology, The University of Lahore, Gujrat Campus, Gujrat, Pakistan

    Fazli Wahid

Authors
  1. Muhammad Aamir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nazri Mohd Nawi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fazli Wahid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hairulnizam Mahdin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fazli Wahid.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aamir, M., Mohd Nawi, N., Wahid, F. et al. A deep contractive autoencoder for solving multiclass classification problems. Evol. Intel. 14, 1619–1633 (2021). https://doi.org/10.1007/s12065-020-00424-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12065-020-00424-6

Keywords

Search

Navigation