Advertisement

Assessment of Autoencoder Architectures for Data Representation

  • Karishma PawarEmail author
  • Vahida Z. AttarEmail author
Chapter
First Online:
Part of the Studies in Computational Intelligence book series (SCI, volume 866)

Abstract

Efficient representation learning of data distribution is part and parcel of successful execution of any machine learning based model. Autoencoders are good at learning the representation of data with lower dimensions. Traditionally, autoencoders have been widely used for data compression in order to represent the structural data. Data compression is one of the most important tasks in applications based on Computer Vision, Information Retrieval, Natural Language Processing, etc. The aim of data compression is to convert the input data into smaller representation retaining the quality of input data. Many lossy and lossless data compression techniques like Flate/deflate compression, Lempel–Ziv–Welch compression, Huffman compression, Run-length encoding compression, JPEG compression are available. Similarly, autoencoders are unsupervised neural networks used for representing the structural data by data compression. Due to wide availability of high-end processing chips and large datasets, deep learning has gained a lot attention from academia, industries and research centers to solve multitude of problems. Considering the state-of-the-art literature, autoencoders are widely used architectures in many deep learning applications for representation and manifold learning and serve as popular option for dimensionality reduction. Therefore, this chapter aims to shed light upon applicability of variants of autoencoders to multiple application domains. In this chapter, basic architecture and variants of autoencoder viz. Convolutional autoencoder, Variational autoencoder, Sparse autoencoder, stacked autoencoder, Deep autoencoder, to name a few, have been thoroughly studied. How the layer size and depth of deep autoencoder model affect the overall performance of the system has also been discussed. We also outlined the suitability of various autoencoder architectures to different application areas. This would help the research community to choose the suitable autoencoder architecture for the problem to be solved.

Keywords

Autoencoders Deep learning Dimensionality reduction Representation learning Data representation 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013)CrossRefGoogle Scholar
  2. 2.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar
  3. 3.
    Pathak, A.R., Pandey, M., Rautaray, S.: Adaptive framework for deep learning based dynamic and temporal topic modeling from big data. Recent Pat. Eng. 13, 1 (2019).  https://doi.org/10.2174/1872212113666190329234812CrossRefGoogle Scholar
  4. 4.
    Pathak, A.R., Pandey, M., Rautaray, S.: Adaptive model for dynamic and temporal topic modeling from big data using deep learning architecture. Int. J. Intell. Syst. Appl. 11(6), 13–27 (MECS-Press)Google Scholar
  5. 5.
    Pathak, A.R., Pandey, M., Rautaray, S., Pawar, K.: Assessment of object detection using deep convolutional neural networks. In: Bhalla, S., Bhateja, V., Chandavale, A.A., Hiwale, A.S., Satapathy, S.C. (eds.) Intelligent Computing and Information and Communication, pp. 457–466. Springer Singapore (2018)CrossRefGoogle Scholar
  6. 6.
    Pathak, A.R., Pandey, M., Rautaray, S.: Deep learning approaches for detecting objects from images: a review. In: Pattnaik, P.K., Rautaray, S.S., Das, H., Nayak, J. (eds.) Progress in Computing, Analytics and Networking, pp. 491–499. Springer Singapore (2018)Google Scholar
  7. 7.
    Pathak, A.R., Pandey, M., Rautaray, S.: Application of deep learning for object detection. Procedia Comput. Sci. 132, 1706–1717 (2018)CrossRefGoogle Scholar
  8. 8.
    Pawar, K., Attar, V.: Deep learning approaches for video-based anomalous activity detection. World Wide Web 22, 571–601 (2019)CrossRefGoogle Scholar
  9. 9.
    Pawar, K., Attar, V.: Deep Learning approach for detection of anomalous activities from surveillance videos. In: CCIS. Springer (2019, in Press)Google Scholar
  10. 10.
    Khare, K., Darekar, O., Gupta, P., Attar, V.Z.: Short term stock price prediction using deep learning. In: 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), pp. 482–486 (2017)Google Scholar
  11. 11.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Kurtz, K.J.: The divergent autoencoder (DIVA) model of category learning. Psychon. Bull. Rev. 14, 560–576 (2007)CrossRefGoogle Scholar
  13. 13.
    Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill (2016).  https://doi.org/10.23915/distill.00003
  14. 14.
    Zhang, Z., et al: Depth-based subgraph convolutional auto-encoder for network representation learning. Pattern Recognit. (2019)Google Scholar
  15. 15.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014). http://arxiv.org/abs/1406.1078
  16. 16.
    Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using LSTMs. In: International Conference on Machine Learning, pp. 843–852 (2015)Google Scholar
  17. 17.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)Google Scholar
  18. 18.
    Poultney, C., Chopra, S., Cun, Y.L., et al.: Efficient learning of sparse representations with an energy-based model. In: Advances in Neural Information Processing Systems, pp. 1137–1144 (2007)Google Scholar
  19. 19.
    Lee, H., Ekanadham, C., Ng, A.Y.: Sparse deep belief net model for visual area V2. In: Advances in Neural Information Processing Systems, pp. 873–880 (2008)Google Scholar
  20. 20.
    Zou, W.Y., Ng, A.Y., Yu, K.: Unsupervised learning of visual invariance with temporal coherence. In: NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning, vol. 3 (2011)Google Scholar
  21. 21.
    Jiang, X., Zhang, Y., Zhang, W., Xiao, X.: A novel sparse auto-encoder for deep unsupervised learning. In 2013 Sixth International Conference on Advanced Computational Intelligence (ICACI), pp. 256–261 (2013)Google Scholar
  22. 22.
    Le, Q.V., et al.: Building high-level features using large scale unsupervised learning (2011). http://arxiv.org/abs/1112.6209
  23. 23.
    Chen, J., et al.: Cross-covariance regularized autoencoders for nonredundant sparse feature representation. Neurocomputing 316, 49–58 (2018)CrossRefGoogle Scholar
  24. 24.
    Goroshin, R., LeCun, Y.: Saturating auto-encoders (2013). http://arxiv.org/abs/1301.3577
  25. 25.
    Liu, W., Ma, T., Tao, D., You, J.H.S.A.E.: A Hessian regularized sparse auto-encoders. Neurocomputing 187, 59–65 (2016)CrossRefGoogle Scholar
  26. 26.
    Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 833–840 (2011)Google Scholar
  27. 27.
    Rifai, S., et al.: Higher order contractive auto-encoder. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 645–660 (2011)CrossRefGoogle Scholar
  28. 28.
    Alain, G., Bengio, Y.: What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res. 15, 3563–3593 (2014)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Mesnil, G., et al.: Unsupervised and transfer learning challenge: a deep learning approach. In: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning Workshop, vol. 27, pp. 97–111 (2011)Google Scholar
  30. 30.
    Konda, K., Memisevic, R., Krueger, D.: Zero-bias autoencoders and the benefits of co-adapting features (2014). http://arxiv.org/abs/1402.3337
  31. 31.
    Makhzani, A., Frey, B.: K-sparse autoencoders (2013). http://arxiv.org/abs/1312.5663
  32. 32.
    Makhzani, A., Frey, B.J.: Winner-take-all autoencoders. In: Advances in Neural Information Processing Systems, pp. 2791–2799 (2015)Google Scholar
  33. 33.
    Ng, A.: Sparse Autoencoder. CS294A Lecture Notes, vol. 72, pp. 1–19 (2011)Google Scholar
  34. 34.
    Liang, K., Chang, H., Cui, Z., Shan, S., Chen, X.: Representation learning with smooth autoencoder. In: Asian Conference on Computer Vision, pp. 72–86 (2014)CrossRefGoogle Scholar
  35. 35.
    Kampffmeyer, M., Løkse, S., Bianchi, F.M., Jenssen, R., Livi, L.: The deep kernelized autoencoder. Appl. Soft Comput. 71, 816–825 (2018)CrossRefGoogle Scholar
  36. 36.
    Majumdar, A.: Graph structured autoencoder. Neural Netw. 106, 271–280 (2018)CrossRefGoogle Scholar
  37. 37.
    Sankaran, A., Vatsa, M., Singh, R., Majumdar, A.: Group sparse autoencoder. Image Vis. Comput. 60, 64–74 (2017)CrossRefGoogle Scholar
  38. 38.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103 (2008)Google Scholar
  39. 39.
    Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Ferles, C., Papanikolaou, Y., Naidoo, K.J.: Denoising autoencoder self-organizing map (DASOM). Neural Netw. 105, 112–131 (2018)CrossRefGoogle Scholar
  41. 41.
    Chen, M., Weinberger, K., Sha, F., Bengio, Y.: Marginalized denoising auto-encoders for nonlinear representations. In: International Conference on Machine Learning, pp. 1476–1484 (2014)Google Scholar
  42. 42.
    Maheshwari, S., Majumdar, A.: Hierarchical autoencoder for collaborative filtering. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–7 (2018)Google Scholar
  43. 43.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). http://arxiv.org/abs/1312.6114
  44. 44.
    Burda, Y., Grosse, R., Salakhutdinov, R.: Importance weighted autoencoders (2015). http://arxiv.org/abs/1509.00519
  45. 45.
    Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders (2015). http://arxiv.org/abs/1511.05644
  46. 46.
    Wang, X., Peng, D., Hu, P., Sang, Y.: Adversarial correlated autoencoder for unsupervised multi-view representation learning. Knowl. Based Syst. (2019)Google Scholar
  47. 47.
    Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders (2017). http://arxiv.org/abs/1711.01558
  48. 48.
    Kim, Y., Zhang, K., Rush, A.M., LeCun, Y., et al.: Adversarially regularized autoencoders (2017). http://arxiv.org/abs/1706.04223
  49. 49.
    Yan, X., Chang, H., Shan, S., Chen, X.: Modeling video dynamics with deep dynencoder. In: European Conference on Computer Vision, pp. 215–230 (2014)CrossRefGoogle Scholar
  50. 50.
    Zhao, J., Mathieu, M., Goroshin, R., Lecun, Y.: Stacked what-where auto-encoders (2015). http://arxiv.org/abs/1506.02351
  51. 51.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)CrossRefGoogle Scholar
  52. 52.
    Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Conference on Computer Vision and Pattern Recognition, pp. 2528–2535. IEEE (2010)Google Scholar
  53. 53.
    Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat., 400–407 (1951)MathSciNetzbMATHCrossRefGoogle Scholar
  54. 54.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). http://arxiv.org/abs/1412.6980
  55. 55.
    Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)MathSciNetzbMATHGoogle Scholar
  56. 56.
    Le, Q.V., et al.: On optimization methods for deep learning. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 265–272 (2011)Google Scholar
  57. 57.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J., et al.: Learning representations by back-propagating errors. Cogn. Model. 5, 1 (1988)Google Scholar
  58. 58.
    Hinton, G.E., McClelland, J.L.: Learning representations by recirculation. In: Neural Information Processing Systems, pp. 358–366 (1988)Google Scholar
  59. 59.
    Zhou, Y., Arpit, D., Nwogu, I., Govindaraju, V.: Is joint training better for deep auto-encoders? (2014). http://arxiv.org/abs/1405.1380
  60. 60.
    Qi, Y., Wang, Y., Zheng, X., Wu, Z.: Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6716–6720 (2014)Google Scholar
  61. 61.
    Kukačka, J., Golkov, V., Cremers, D.: Regularization for deep learning: a taxonomy (2017). http://arxiv.org/abs/1710.10686
  62. 62.
    Lamb, A., Dumoulin, V., Courville, A.: Discriminative regularization for generative models (2016). http://arxiv.org/abs/1602.03220
  63. 63.
    Kamyshanska, H., Memisevic, R.: The potential energy of an autoencoder. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1261–1273 (2015)CrossRefGoogle Scholar
  64. 64.
    Kamyshanska, H., Memisevic, R.: On autoencoder scoring. In: International Conference on Machine Learning, pp. 720–728 (2013)Google Scholar
  65. 65.
    Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Advances in Neural Information Processing Systems, pp. 950–957 (1992)Google Scholar
  66. 66.
    Fan, Y.J.: Autoencoder node saliency: selecting relevant latent representations. Pattern Recognit. 88, 643–653 (2019)CrossRefGoogle Scholar
  67. 67.
    LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp 9–48. Springer (2012)Google Scholar
  68. 68.
    Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Advances in Neural Information Processing Systems, pp. 971–980 (2017)Google Scholar
  69. 69.
    Leonard, M.: Deep Learning Nanodegree Foundation Course. Lecture Notes in Autoencoders. Udacity (2018)Google Scholar
  70. 70.
    Xiong, Y., Zuo, R.: Recognition of geochemical anomalies using a deep autoencoder network. Comput. Geosci. 86, 75–82 (2016)CrossRefGoogle Scholar
  71. 71.
    Leng, B., Guo, S., Zhang, X., Xiong, Z.: 3D object retrieval with stacked local convolutional autoencoder. Sig. Process. 112, 119–128 (2015)CrossRefGoogle Scholar
  72. 72.
    Ribeiro, M., Lazzaretti, A.E., Lopes, H.S.: A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recognit. Lett. 105, 13–22 (2018)CrossRefGoogle Scholar
  73. 73.
    Li, L., Li, X., Yang, Y., Dong, J.: Indoor tracking trajectory data similarity analysis with a deep convolutional autoencoder. Sustain. Cities Soc. 45, 588–595 (2019)CrossRefGoogle Scholar
  74. 74.
    Wan, X., Zhao, C., Wang, Y., Liu, W.: Stacked sparse autoencoder in hyperspectral data classification using spectral-spatial, higher order statistics and multifractal spectrum features. Infrared Phys. Technol. 86, 77–89 (2017)CrossRefGoogle Scholar
  75. 75.
    McCoy, J.T., Kroon, S., Auret, L.: Variational autoencoders for missing data imputation with application to a simulated milling circuit. IFAC PapersOnLine 51, 141–146 (2018)CrossRefGoogle Scholar
  76. 76.
    Wu, C., et al.: Semi-supervised dimensional sentiment analysis with variational autoencoder. Knowl. Based Syst. 165, 30–39 (2019)CrossRefGoogle Scholar
  77. 77.
    Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: European Conference on Computer Vision, pp. 1–16 (2014)Google Scholar
  78. 78.
    Masci, J., Meier, U., Cirecsan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks, pp. 52–59 (2011)Google Scholar
  79. 79.
    Liou, C.-Y., Cheng, W.-C., Liou, J.-W., Liou, D.-R.: Autoencoder for words. Neurocomputing 139, 84–96 (2014)CrossRefGoogle Scholar
  80. 80.
    Carreira-Perpinan, M.A., Raziperchikolaei, R.: Hashing with binary autoencoders. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  81. 81.
    Pan, S., et al.: Adversarially regularized graph autoencoder for graph embedding (2018). http://arxiv.org/abs/1802.04407
  82. 82.
    Li, M., et al.: GRAINS: generative recursive autoencoders for INdoor scenes. ACM Trans. Graph. 38, 12:1–12:16 (2019)CrossRefGoogle Scholar
  83. 83.
    Alaverdyan, Z., Chai, J., Lartizien, C.: Unsupervised feature learning for outlier detection with stacked convolutional autoencoders, siamese networks and wasserstein autoencoders: application to epilepsy detection. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 210–217. Springer (2018)Google Scholar
  84. 84.
    Hou, L., et al.: Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images. Pattern Recognit. 86, 188–200 (2019)CrossRefGoogle Scholar
  85. 85.
    Ullah, A., Muhammad, K., Haq, I.U., Baik, S.W.: Action recognition using optimized deep autoencoder and CNN for surveillance data streams of non-stationary environments. Futur. Gener. Comput. Syst. (2019)Google Scholar
  86. 86.
    Zhao, C., Zhang, L.: Spectral-spatial stacked autoencoders based on low-rank and sparse matrix decomposition for hyperspectral anomaly detection. Infrared Phys. Technol. 92, 166–176 (2018)CrossRefGoogle Scholar
  87. 87.
    Singh, M., Nagpal, S., Vatsa, M., Singh, R.: Are you eligible? Predicting adulthood from face images via class specific mean autoencoder. Pattern Recognit. Lett. 119, 121–130 (2019)CrossRefGoogle Scholar
  88. 88.
    Tasnim, S., Rahman, A., Oo, A.M.T., Haque, M.E.: Autoencoder for wind power prediction. Renewables Wind. Water Sol. 4, 6 (2017)Google Scholar
  89. 89.
    Lv, S.-X., Peng, L., Wang, L.: Stacked autoencoder with echo-state regression for tourism demand forecasting using search query data. Appl. Soft Comput. 73, 119–133 (2018)CrossRefGoogle Scholar
  90. 90.
    Xie, R., Wen, J., Quitadamo, A., Cheng, J., Shi, X.: A deep auto-encoder model for gene expression prediction. BMC Genom. 18, 845 (2017)CrossRefGoogle Scholar
  91. 91.
    Zhang, J., Li, K., Liang, Y., Li, N.: Learning 3D faces from 2D images via stacked contractive autoencoder. Neurocomputing 257, 67–78 (2017)CrossRefGoogle Scholar
  92. 92.
    Gareis, I.E., Vignolo, L.D., Spies, R.D., Rufiner, H.L.: Coherent averaging estimation autoencoders applied to evoked potentials processing. Neurocomputing 240, 47–58 (2017)CrossRefGoogle Scholar
  93. 93.
    Mehta, J., Majumdar, A.: RODEO: robust DE-aliasing autoencoder for real-time medical image reconstruction. Pattern Recognit. 63, 499–510 (2017)CrossRefGoogle Scholar
  94. 94.
    Liu, Y., Feng, X., Zhou, Z.: Multimodal video classification with stacked contractive autoencoders. Sig. Process. 120, 761–766 (2016)CrossRefGoogle Scholar
  95. 95.
    Zhang, Z., et al.: Deep neural network-based bottleneck feature and denoising autoencoder-based dereverberation for distant-talking speaker identification. EURASIP J. Audio Speech Music Process. 2015, 12 (2015)Google Scholar
  96. 96.
    Makkie, M., Huang, H., Zhao, Y., Vasilakos, A.V., Liu, T.: Fast and scalable distributed deep convolutional autoencoder for fMRI big data analytics. Neurocomputing 325, 20–30 (2019)CrossRefGoogle Scholar
  97. 97.
    Guo, Q., et al.: Learning robust uniform features for cross-media social data by using cross autoencoders. Knowl. Based Syst. 102, 64–75 (2016)CrossRefGoogle Scholar
  98. 98.
    Su, J., et al.: A neural generative autoencoder for bilingual word embeddings. Inf. Sci. (Ny) 424, 287–300 (2018)MathSciNetCrossRefGoogle Scholar
  99. 99.
    Gianniotis, N., Kügler, S.D., Tino, P., Polsterer, K.L.: Model-coupled autoencoder for time series visualization. Neurocomputing 192, 139–146 (2016)CrossRefGoogle Scholar
  100. 100.
    Hwang, U., Park, J., Jang, H., Yoon, S., Cho, N.I.: PuVAE: a variational autoencoder to purify adversarial examples (2019). http://arxiv.org/abs/1903.00585CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer Engineering & ITCollege of Engineering Pune (COEP)PuneIndia

Personalised recommendations

Cite chapter

Buy options