Skip to main content

Hybrid Contractive Auto-encoder with Restricted Boltzmann Machine For Multiclass Classification

Arabian Journal for Science and Engineering volume 46, pages 9237–9251 (2021)Cite this article

Abstract

Contractive auto-encoder (CAE) is a type of auto-encoders and a deep learning algorithm that is based on multilayer training approach. It is considered as one of the most powerful, efficient and robust classification techniques, more specifically feature reduction. The problem independence, easy implementation and intelligence of solving sophisticated problems make it distinct from other deep learning approaches. However, CAE fails in data dimensionality reduction that cause difficulty to capture the useful information within the features space. In order to resolve the issues of CAE, restricted Boltzmann machine (RBM) layers have been integrated with CAE to enhance the dimensionality reduction and a randomized factor for hidden layer parameters. The proposed model has been evaluated on four benchmark variant datasets of MNIST. The results have been compared with four well-known multiclass class classification approaches including standard CAE, RBM, AlexNet and artificial neural network. A considerable amount of improvement has been observed in the performance of proposed model as compared to other classification techniques. The proposed CAE–RBM showed an improvement of 2–4% on MNIST(basic), 9–12% for MNIST(rot), 7–12% for MNIST(bg-rand) and 7–10% for MNIST(bg-img) dataset in term of final accuracy.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Abbreviations

AE:

Auto-encoder

RBM:

Restricted Boltzmann machine

CAE:

Contractive auto-encoder

CAE–RBM:

Hybrid contractive auto-encoder–restricted Boltzmann machine

ANN:

Artificial neural network

SVM:

Support vector machine

kNN:

k-Nearest neighbor

CNN:

Convolution neural network

DL:

Deep learning

ML:

Machine learning

ROC:

Receiver operating characteristic

CM:

Confusion matrix

MNIST:

Modified National Institute of Standards and Technology (database)

rot:

MNIST random rotation digits

bg-rand:

Random noise background digits

bg-img:

Random background digits

References

  1. Aamir, M.; Nawi, N.M.; Shahzad, A.; Mahdin, H.; Rehman, M.Z.: A new argumentative based reasoning framework with rough set for decision making. In: 2017 6th ICT International Student Project Conference (ICT-ISPC), pp. 1–4, May (2017)

  2. Aamir, M.; Nawi, N.M.; Wahid, F.; Mahdin, H.: A deep contractive autoencoder for solving multiclass classification problems. In: Evolutionary Intelligence, pp. 1–15 (2020).

  3. Aamir, M.; Wahid, F.; Mahdin, H.; Nawi, N.M.: An efficient normalized restricted boltzmann machine for solving multiclass classification problems. Int. J. Adv. Comput. Sci. Appl. 10, 8 (2019)

    Google Scholar 

  4. Aamir, M.; Nawi, N.M.; Mahdin, H.B.; Naseem, R.; Zulqarnain, M.: Auto-encoder variants for solving handwritten digits classification problem. Int. J. Fuzzy Logic Intell. Syst. 20(1), 8–16 (2020)

    Article  Google Scholar 

  5. Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M. et al.: Tensorflow: A system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 16), pp. 265–283 (2016).

  6. Aldwairi, T.; Perera, D.; Novotny, M.A.: An evaluation of the performance of restricted boltzmann machines as a model for anomaly network intrusion detection. Comput. Netw. 144, 111–119 (2018)

    Article  Google Scholar 

  7. Ayesha, S.; Hanif, M.K.; Talib, R.: Overview and comparative study of dimensionality reduction techniques for high dimensional data. Inf. Fusion 59, 44–58 (2020)

    Article  Google Scholar 

  8. Buda, M.; Maki, A.; Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 106, 249–259 (2018)

    Article  Google Scholar 

  9. Cai, W.; Wei, Z.: Remote sensing image classification based on a cross-attention mechanism and graph convolution. In: IEEE Geoscience and Remote Sensing Letters, pp. 1–5 (2020.

  10. Chen, M.; Weinberger, K.Q.; Xu, Z.; Sha, F.: Marginalizing stacked linear denoising autoencoders. J. Mach. Learn. Res. 16(1), 3849–3875 (2015)

    MathSciNet  Google Scholar 

  11. Dehban, A.; Jamone, L.; Kampff, A.R.; Santos-Victor, J.: Denoising auto-encoders for learning of objects and tools affordances in continuous space. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 4866–4871, IEEE (2016)

  12. Druzhkov, P.; Kustikova, V.: A survey of deep learning methods and software tools for image classification and object detection. Pattern Recogn. Image Anal. 26(1), 9–15 (2016)

    Article  Google Scholar 

  13. Goodfellow, I.; Bengio, Y.; Courville, A.: Deep Learning. MIT Press, New York (2016)

    MATH  Google Scholar 

  14. Goroshin, R.; LeCun, Y.: Saturating auto-encoders. arXiv preprint arXiv:1301.3577 (2013).

  15. Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al.: Recent advances in convolutional neural networks. Pattern Recogn. 77, 354–377 (2018)

    Article  Google Scholar 

  16. Hinton, G.E.: A practical guide to training restricted boltzmann machines. In: Neural Networks: Tricks of the Trade, pp. 599–619. Springer (2012)

  17. Izonin, I.; Tkachenko, R.; Kryvinska, N.; Tkachenko, P. et al.: Multiple linear regression based on coefficients identification using non-iterative sgtm neural-like structure. In: International Work-Conference on Artificial Neural Networks, pp. 467–479, Springer (2019).

  18. Kang, T.; Chen, P.; Quackenbush, J.; Ding, W.: A novel deep learning model by stacking conditional restricted boltzmann machine and deep neural network. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1316–1324 (2020).

  19. Karras, T.; Aittala, M.; Hellsten, J.; Laine, S.; Lehtinen, J.; Aila, T.: Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676 (2020).

  20. Knyaz, V.A.; Vygolov, O.; Kniaz, V.V.; Vizilter, Y.; Gorbatsevich, V.; Luhmann, T.; Conen, N.: Deep learning of convolutional auto-encoder for image matching and 3d object reconstruction in the infrared range. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2155–2164 (2017)

  21. Konda, K.; Memisevic, R.; Krueger, D.: Zero-bias autoencoders and the benefits of co-adapting features (2014). arXiv preprint arXiv:1402.3337.

  22. Krizhevsky, A.; Sutskever, I.; Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  23. Li, B., Najafi, M.H., Lilja, D.J.: Using stochastic computing to reduce the hardware requirements for a restricted boltzmann machine classifier. In: Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 36–41, ACM (2016)

  24. Liu, L.: Hierarchical learning for large multi-class network classification. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2307–2312, IEEE (2016).

  25. Liu, Y.; Feng, X.; Zhou, Z.: Multimodal video classification with stacked contractive autoencoders. Signal Process. 120, 761–766 (2016)

    Article  Google Scholar 

  26. Long, M.; Zhu, H.; Wang, J.; Jordan, M.I.: Deep transfer learning with joint adaptation networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2208–2217, JMLR. org (2017)

  27. Lv, N.; Chen, C.; Qiu, T.; Sangaiah, A.K.: Deep learning and superpixel feature extraction based on contractive autoencoder for change detection in sar images. IEEE Trans. Ind. Inf. 14(12), 5530–5538 (2018)

    Article  Google Scholar 

  28. Mahmoud, A.M.; Alrowais, F.; Karamti, H.: A hybrid deep contractive autoencoder and restricted boltzmann machine approach to differentiate representation of female brain disorder. Proc. Comput. Sci. 176, 1033–1042 (2020)

    Article  Google Scholar 

  29. Mao, X.; Li, Q.; Xie, H.; Lau, R.Y.; Wang, Z.; Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017).

  30. Meyer, D.; Wien, F.T.: Support vector machines. The Interface to libsvm in package e1071, p. 28 (2015)

  31. Nasiri, J.A.; Charkari, N.M.; Jalili, S.: Least squares twin multi-class classification support vector machine. Pattern Recogn. 48(3), 984–992 (2015)

    Article  Google Scholar 

  32. Nawi, N.M.; Hamzah, F.; Hamid, N.A.; Rehman, M.Z.; Aamir, M.; Ramli, A.A.: An optimized back propagation learning algorithm with adaptive learning rate. Learning 500, 2 (2017)

    Google Scholar 

  33. Ning, X.; Li, W.; Tang, B.; He, H.: Buldp: biomimetic uncorrelated locality discriminant projection for feature extraction in face recognition. IEEE Trans. Image Process. 27(5), 2575–2586 (2018)

    Article  MathSciNet  Google Scholar 

  34. Oyedotun, O.K.; Khashman, A.: Deep learning in vision-based static hand gesture recognition. Neural Comput. Appl. 28(12), 3941–3951 (2017)

    Article  Google Scholar 

  35. Ran, R.; Ren, Y.; Zhang, S.; Fang, B.: A novel discriminant locality preserving projections method. In: Journal of Mathematical Imaging and Vision, pp. 1–14 (2021)

  36. Reddy, G.T.; Reddy, M.P.K.; Lakshmanna, K.; Kaluri, R.; Rajput, D.S.; Srivastava, G.; Baker, T.: Analysis of dimensionality reduction techniques on big data. IEEE Access 8, 54776–54788 (2020)

    Article  Google Scholar 

  37. Rifai, S.; Mesnil, G.; Vincent, P.; Muller, X.; Bengio, Y.; Dauphin, Y.; Glorot, X.: Higher order contractive auto-encoder. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 645–660, Springer (2011).

  38. Rifai, S.; Vincent, P.; Muller, X.; Glorot, X.; Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 833–840, Omnipress (2011).

  39. Russell, S.J.; Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education Limited, Malaysia (2016)

    MATH  Google Scholar 

  40. Sainath, T.N., Mohamed, A.-R., Kingsbury, B., Ramabhadran, B.: Deep convolutional neural networks for lvcsr. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8614–8618, IEEE (2013).

  41. Senan, N.; Aamir, M.; Ibrahim, R.; Taujuddin, N.S.A.M.; Muda, W.W.: An efficient convolutional neural network for paddy leaf disease and pest classification. Int. J. Adv. Comput. Sci. Appl. 11, 7 (2020)

    Google Scholar 

  42. Sengan, S.; Arokia Jesu Prabhu, L.; Ramachandran, V.; Priya, V.; Ravi, L.; Subramaniyaswamy, V.: Images super-resolution by optimal deep alexnet architecture for medical application: a novel docaln. J. Intell. Fuzzy Syst., no. Preprint, pp. 1–14 (2020).

  43. Sharma, N.; Singh, M.: Modifying naive bayes classifier for multinomial text classification. In: 2016 International Conference on Recent Advances and Innovations in Engineering (ICRAIE), pp. 1–7, IEEE (2016)

  44. Sharma, A.; Tuzel, O.; Jacobs, D.W.: Deep hierarchical parsing for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 530–538 (2015)

  45. Sur, C.: Relgan: generalization of consistency for gan with disjoint constraints and relative learning of generative processes for multiple transformation learning. arXiv preprint arXiv:2006.07809 (2020).

  46. Taherkhani, A.; Cosma, G.; McGinnity, T.M.: Deep-fs: a feature selection algorithm for deep boltzmann machines. Neurocomputing 322, 22–37 (2018)

    Article  Google Scholar 

  47. Tao, M.; Tang, H.; Wu, S.; Sebe, N.; Wu, F.; Jing, X.-Y.: Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.05865 (2020).

  48. Tkachenko, R.; Izonin, I.; Vitynskyi, P.; Lotoshynska, N.; Pavlyuk, O.: Development of the non-iterative supervised learning predictor based on the ito decomposition and sgtm neural-like structure for managing medical insurance costs. Data 3(4), 46 (2018)

    Article  Google Scholar 

  49. Vitynskyi, P.; Tkachenko, R.; Izonin, I.; Kutucu, H.: Hybridization of the sgtm neural-like structure through inputs polynomial extension. In: 2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP), pp. 386–391, IEEE (2018)

  50. Wang, Y.; Yao, H.; Zhao, S.: Auto-encoder based dimensionality reduction. Neurocomputing 184, 232–242 (2016)

    Article  Google Scholar 

  51. Wen, L.; Gao, L.; Li, X.: A new deep transfer learning based on sparse auto-encoder for fault diagnosis. IEEE Trans. Syst. Man Cybernet. Syst. 99, 1–9 (2017)

    Google Scholar 

  52. Xiong, P.; Wang, H.; Liu, M.; Lin, F.; Hou, Z.; Liu, X.: A stacked contractive denoising auto-encoder for ecg signal denoising. Physiol. Measur. 37(12), 2214 (2016)

    Article  Google Scholar 

  53. Young, T.; Hazarika, D.; Poria, S.; Cambria, E.: Recent trends in deep learning based natural language processing. IEEE Comput. Intell. Mag 13(3), 55–75 (2018)

    Article  Google Scholar 

  54. Zhang, H.; Goodfellow, I.; Metaxas, D.; Odena, A.: Self-attention generative adversarial networks. In: International Conference on Machine Learning, pp. 7354–7363, PMLR (2019)

  55. Zhang, W.; Liu, Y.; Dong, C.; Qiao, Y.: Ranksrgan: generative adversarial networks with ranker for image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3096–3105 (2019).

  56. Zhang, S.; Li, X.; Zong, M.; Zhu, X.; Wang, R.: Efficient knn classification with different numbers of nearest neighbors. IEEE Trans. Neural Netw. Learn. Syst. 29(5), 1774–1785 (2017)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thanks Ministry of Education Malaysia, Universiti Tun Hussein Onn Malaysia (UTHM) and University of Derby, United Kingdom, for carrying out this research activity.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn, Parit Raja, Malaysia

    Muhammad Aamir, Nazri Mohd Nawi, M. Z. Rehman & Muhammad Zulqarnain

  2. School of Electronics, Computing and Mathematics, University of Derby, Derby, UK

    Muhammad Aamir & Muhammad Sadiq Hasan Zada

  3. Soft Computing and Data Mining Center, Universiti Tun Hussein Onn, Parit Raja, Malaysia

    Nazri Mohd Nawi

  4. Department of Information Technology, The University of Haripur, Haripur, Khyber Pakhtunkhwa, Pakistan

    Fazli Wahid

Authors
  1. Muhammad Aamir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nazri Mohd Nawi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fazli Wahid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Sadiq Hasan Zada
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. M. Z. Rehman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Muhammad Zulqarnain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Muhammad Aamir or Fazli Wahid.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Aamir, M., Nawi, N.M., Wahid, F. et al. Hybrid Contractive Auto-encoder with Restricted Boltzmann Machine For Multiclass Classification. Arab J Sci Eng 46, 9237–9251 (2021). https://doi.org/10.1007/s13369-021-05674-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-021-05674-9

Keywords

  • Contractive auto-encoder
  • Restricted Boltzmann machine
  • Classification
  • Mnist variants