Abstract
Efficient representation learning of data distribution is part and parcel of successful execution of any machine learning based model. Autoencoders are good at learning the representation of data with lower dimensions. Traditionally, autoencoders have been widely used for data compression in order to represent the structural data. Data compression is one of the most important tasks in applications based on Computer Vision, Information Retrieval, Natural Language Processing , etc. The aim of data compression is to convert the input data into smaller representation retaining the quality of input data. Many lossy and lossless data compression techniques like Flate/deflate compression, Lempel–Ziv–Welch compression, Huffman compression, Run-length encoding compression, JPEG compression are available. Similarly, autoencoders are unsupervised neural networks used for representing the structural data by data compression. Due to wide availability of high-end processing chips and large datasets, deep learning has gained a lot attention from academia, industries and research centers to solve multitude of problems. Considering the state-of-the-art literature, autoencoders are widely used architectures in many deep learning applications for representation and manifold learning and serve as popular option for dimensionality reduction . Therefore, this chapter aims to shed light upon applicability of variants of autoencoders to multiple application domains. In this chapter, basic architecture and variants of autoencoder viz. Convolutional autoencoder, Variational autoencoder, Sparse autoencoder, stacked autoencoder, Deep autoencoder , to name a few, have been thoroughly studied. How the layer size and depth of deep autoencoder model affect the overall performance of the system has also been discussed. We also outlined the suitability of various autoencoder architectures to different application areas. This would help the research community to choose the suitable autoencoder architecture for the problem to be solved.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)
Pathak, A.R., Pandey, M., Rautaray, S.: Adaptive framework for deep learning based dynamic and temporal topic modeling from big data. Recent Pat. Eng. 13, 1 (2019). https://doi.org/10.2174/1872212113666190329234812
Pathak, A.R., Pandey, M., Rautaray, S.: Adaptive model for dynamic and temporal topic modeling from big data using deep learning architecture. Int. J. Intell. Syst. Appl. 11(6), 13–27 (MECS-Press)
Pathak, A.R., Pandey, M., Rautaray, S., Pawar, K.: Assessment of object detection using deep convolutional neural networks. In: Bhalla, S., Bhateja, V., Chandavale, A.A., Hiwale, A.S., Satapathy, S.C. (eds.) Intelligent Computing and Information and Communication, pp. 457–466. Springer Singapore (2018)
Pathak, A.R., Pandey, M., Rautaray, S.: Deep learning approaches for detecting objects from images: a review. In: Pattnaik, P.K., Rautaray, S.S., Das, H., Nayak, J. (eds.) Progress in Computing, Analytics and Networking, pp. 491–499. Springer Singapore (2018)
Pathak, A.R., Pandey, M., Rautaray, S.: Application of deep learning for object detection. Procedia Comput. Sci. 132, 1706–1717 (2018)
Pawar, K., Attar, V.: Deep learning approaches for video-based anomalous activity detection. World Wide Web 22, 571–601 (2019)
Pawar, K., Attar, V.: Deep Learning approach for detection of anomalous activities from surveillance videos. In: CCIS. Springer (2019, in Press)
Khare, K., Darekar, O., Gupta, P., Attar, V.Z.: Short term stock price prediction using deep learning. In: 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), pp. 482–486 (2017)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)
Kurtz, K.J.: The divergent autoencoder (DIVA) model of category learning. Psychon. Bull. Rev. 14, 560–576 (2007)
Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill (2016). https://doi.org/10.23915/distill.00003
Zhang, Z., et al: Depth-based subgraph convolutional auto-encoder for network representation learning. Pattern Recognit. (2019)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014). http://arxiv.org/abs/1406.1078
Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using LSTMs. In: International Conference on Machine Learning, pp. 843–852 (2015)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)
Poultney, C., Chopra, S., Cun, Y.L., et al.: Efficient learning of sparse representations with an energy-based model. In: Advances in Neural Information Processing Systems, pp. 1137–1144 (2007)
Lee, H., Ekanadham, C., Ng, A.Y.: Sparse deep belief net model for visual area V2. In: Advances in Neural Information Processing Systems, pp. 873–880 (2008)
Zou, W.Y., Ng, A.Y., Yu, K.: Unsupervised learning of visual invariance with temporal coherence. In: NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning, vol. 3 (2011)
Jiang, X., Zhang, Y., Zhang, W., Xiao, X.: A novel sparse auto-encoder for deep unsupervised learning. In 2013 Sixth International Conference on Advanced Computational Intelligence (ICACI), pp. 256–261 (2013)
Le, Q.V., et al.: Building high-level features using large scale unsupervised learning (2011). http://arxiv.org/abs/1112.6209
Chen, J., et al.: Cross-covariance regularized autoencoders for nonredundant sparse feature representation. Neurocomputing 316, 49–58 (2018)
Goroshin, R., LeCun, Y.: Saturating auto-encoders (2013). http://arxiv.org/abs/1301.3577
Liu, W., Ma, T., Tao, D., You, J.H.S.A.E.: A Hessian regularized sparse auto-encoders. Neurocomputing 187, 59–65 (2016)
Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 833–840 (2011)
Rifai, S., et al.: Higher order contractive auto-encoder. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 645–660 (2011)
Alain, G., Bengio, Y.: What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res. 15, 3563–3593 (2014)
Mesnil, G., et al.: Unsupervised and transfer learning challenge: a deep learning approach. In: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning Workshop, vol. 27, pp. 97–111 (2011)
Konda, K., Memisevic, R., Krueger, D.: Zero-bias autoencoders and the benefits of co-adapting features (2014). http://arxiv.org/abs/1402.3337
Makhzani, A., Frey, B.: K-sparse autoencoders (2013). http://arxiv.org/abs/1312.5663
Makhzani, A., Frey, B.J.: Winner-take-all autoencoders. In: Advances in Neural Information Processing Systems, pp. 2791–2799 (2015)
Ng, A.: Sparse Autoencoder. CS294A Lecture Notes, vol. 72, pp. 1–19 (2011)
Liang, K., Chang, H., Cui, Z., Shan, S., Chen, X.: Representation learning with smooth autoencoder. In: Asian Conference on Computer Vision, pp. 72–86 (2014)
Kampffmeyer, M., Løkse, S., Bianchi, F.M., Jenssen, R., Livi, L.: The deep kernelized autoencoder. Appl. Soft Comput. 71, 816–825 (2018)
Majumdar, A.: Graph structured autoencoder. Neural Netw. 106, 271–280 (2018)
Sankaran, A., Vatsa, M., Singh, R., Majumdar, A.: Group sparse autoencoder. Image Vis. Comput. 60, 64–74 (2017)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103 (2008)
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)
Ferles, C., Papanikolaou, Y., Naidoo, K.J.: Denoising autoencoder self-organizing map (DASOM). Neural Netw. 105, 112–131 (2018)
Chen, M., Weinberger, K., Sha, F., Bengio, Y.: Marginalized denoising auto-encoders for nonlinear representations. In: International Conference on Machine Learning, pp. 1476–1484 (2014)
Maheshwari, S., Majumdar, A.: Hierarchical autoencoder for collaborative filtering. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–7 (2018)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). http://arxiv.org/abs/1312.6114
Burda, Y., Grosse, R., Salakhutdinov, R.: Importance weighted autoencoders (2015). http://arxiv.org/abs/1509.00519
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders (2015). http://arxiv.org/abs/1511.05644
Wang, X., Peng, D., Hu, P., Sang, Y.: Adversarial correlated autoencoder for unsupervised multi-view representation learning. Knowl. Based Syst. (2019)
Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders (2017). http://arxiv.org/abs/1711.01558
Kim, Y., Zhang, K., Rush, A.M., LeCun, Y., et al.: Adversarially regularized autoencoders (2017). http://arxiv.org/abs/1706.04223
Yan, X., Chang, H., Shan, S., Chen, X.: Modeling video dynamics with deep dynencoder. In: European Conference on Computer Vision, pp. 215–230 (2014)
Zhao, J., Mathieu, M., Goroshin, R., Lecun, Y.: Stacked what-where auto-encoders (2015). http://arxiv.org/abs/1506.02351
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)
Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Conference on Computer Vision and Pattern Recognition, pp. 2528–2535. IEEE (2010)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat., 400–407 (1951)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). http://arxiv.org/abs/1412.6980
Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)
Le, Q.V., et al.: On optimization methods for deep learning. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 265–272 (2011)
Rumelhart, D.E., Hinton, G.E., Williams, R.J., et al.: Learning representations by back-propagating errors. Cogn. Model. 5, 1 (1988)
Hinton, G.E., McClelland, J.L.: Learning representations by recirculation. In: Neural Information Processing Systems, pp. 358–366 (1988)
Zhou, Y., Arpit, D., Nwogu, I., Govindaraju, V.: Is joint training better for deep auto-encoders? (2014). http://arxiv.org/abs/1405.1380
Qi, Y., Wang, Y., Zheng, X., Wu, Z.: Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6716–6720 (2014)
KukaÄŤka, J., Golkov, V., Cremers, D.: Regularization for deep learning: a taxonomy (2017). http://arxiv.org/abs/1710.10686
Lamb, A., Dumoulin, V., Courville, A.: Discriminative regularization for generative models (2016). http://arxiv.org/abs/1602.03220
Kamyshanska, H., Memisevic, R.: The potential energy of an autoencoder. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1261–1273 (2015)
Kamyshanska, H., Memisevic, R.: On autoencoder scoring. In: International Conference on Machine Learning, pp. 720–728 (2013)
Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Advances in Neural Information Processing Systems, pp. 950–957 (1992)
Fan, Y.J.: Autoencoder node saliency: selecting relevant latent representations. Pattern Recognit. 88, 643–653 (2019)
LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp 9–48. Springer (2012)
Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Advances in Neural Information Processing Systems, pp. 971–980 (2017)
Leonard, M.: Deep Learning Nanodegree Foundation Course. Lecture Notes in Autoencoders. Udacity (2018)
Xiong, Y., Zuo, R.: Recognition of geochemical anomalies using a deep autoencoder network. Comput. Geosci. 86, 75–82 (2016)
Leng, B., Guo, S., Zhang, X., Xiong, Z.: 3D object retrieval with stacked local convolutional autoencoder. Sig. Process. 112, 119–128 (2015)
Ribeiro, M., Lazzaretti, A.E., Lopes, H.S.: A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recognit. Lett. 105, 13–22 (2018)
Li, L., Li, X., Yang, Y., Dong, J.: Indoor tracking trajectory data similarity analysis with a deep convolutional autoencoder. Sustain. Cities Soc. 45, 588–595 (2019)
Wan, X., Zhao, C., Wang, Y., Liu, W.: Stacked sparse autoencoder in hyperspectral data classification using spectral-spatial, higher order statistics and multifractal spectrum features. Infrared Phys. Technol. 86, 77–89 (2017)
McCoy, J.T., Kroon, S., Auret, L.: Variational autoencoders for missing data imputation with application to a simulated milling circuit. IFAC PapersOnLine 51, 141–146 (2018)
Wu, C., et al.: Semi-supervised dimensional sentiment analysis with variational autoencoder. Knowl. Based Syst. 165, 30–39 (2019)
Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: European Conference on Computer Vision, pp. 1–16 (2014)
Masci, J., Meier, U., Cirecsan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks, pp. 52–59 (2011)
Liou, C.-Y., Cheng, W.-C., Liou, J.-W., Liou, D.-R.: Autoencoder for words. Neurocomputing 139, 84–96 (2014)
Carreira-Perpinan, M.A., Raziperchikolaei, R.: Hashing with binary autoencoders. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Pan, S., et al.: Adversarially regularized graph autoencoder for graph embedding (2018). http://arxiv.org/abs/1802.04407
Li, M., et al.: GRAINS: generative recursive autoencoders for INdoor scenes. ACM Trans. Graph. 38, 12:1–12:16 (2019)
Alaverdyan, Z., Chai, J., Lartizien, C.: Unsupervised feature learning for outlier detection with stacked convolutional autoencoders, siamese networks and wasserstein autoencoders: application to epilepsy detection. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 210–217. Springer (2018)
Hou, L., et al.: Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images. Pattern Recognit. 86, 188–200 (2019)
Ullah, A., Muhammad, K., Haq, I.U., Baik, S.W.: Action recognition using optimized deep autoencoder and CNN for surveillance data streams of non-stationary environments. Futur. Gener. Comput. Syst. (2019)
Zhao, C., Zhang, L.: Spectral-spatial stacked autoencoders based on low-rank and sparse matrix decomposition for hyperspectral anomaly detection. Infrared Phys. Technol. 92, 166–176 (2018)
Singh, M., Nagpal, S., Vatsa, M., Singh, R.: Are you eligible? Predicting adulthood from face images via class specific mean autoencoder. Pattern Recognit. Lett. 119, 121–130 (2019)
Tasnim, S., Rahman, A., Oo, A.M.T., Haque, M.E.: Autoencoder for wind power prediction. Renewables Wind. Water Sol. 4, 6 (2017)
Lv, S.-X., Peng, L., Wang, L.: Stacked autoencoder with echo-state regression for tourism demand forecasting using search query data. Appl. Soft Comput. 73, 119–133 (2018)
Xie, R., Wen, J., Quitadamo, A., Cheng, J., Shi, X.: A deep auto-encoder model for gene expression prediction. BMC Genom. 18, 845 (2017)
Zhang, J., Li, K., Liang, Y., Li, N.: Learning 3D faces from 2D images via stacked contractive autoencoder. Neurocomputing 257, 67–78 (2017)
Gareis, I.E., Vignolo, L.D., Spies, R.D., Rufiner, H.L.: Coherent averaging estimation autoencoders applied to evoked potentials processing. Neurocomputing 240, 47–58 (2017)
Mehta, J., Majumdar, A.: RODEO: robust DE-aliasing autoencoder for real-time medical image reconstruction. Pattern Recognit. 63, 499–510 (2017)
Liu, Y., Feng, X., Zhou, Z.: Multimodal video classification with stacked contractive autoencoders. Sig. Process. 120, 761–766 (2016)
Zhang, Z., et al.: Deep neural network-based bottleneck feature and denoising autoencoder-based dereverberation for distant-talking speaker identification. EURASIP J. Audio Speech Music Process. 2015, 12 (2015)
Makkie, M., Huang, H., Zhao, Y., Vasilakos, A.V., Liu, T.: Fast and scalable distributed deep convolutional autoencoder for fMRI big data analytics. Neurocomputing 325, 20–30 (2019)
Guo, Q., et al.: Learning robust uniform features for cross-media social data by using cross autoencoders. Knowl. Based Syst. 102, 64–75 (2016)
Su, J., et al.: A neural generative autoencoder for bilingual word embeddings. Inf. Sci. (Ny) 424, 287–300 (2018)
Gianniotis, N., Kügler, S.D., Tino, P., Polsterer, K.L.: Model-coupled autoencoder for time series visualization. Neurocomputing 192, 139–146 (2016)
Hwang, U., Park, J., Jang, H., Yoon, S., Cho, N.I.: PuVAE: a variational autoencoder to purify adversarial examples (2019). http://arxiv.org/abs/1903.00585
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Appendix
Appendix
List of abbreviations used in this chapter are mentioned in Table 2.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Pawar, K., Attar, V.Z. (2020). Assessment of Autoencoder Architectures for Data Representation. In: Pedrycz, W., Chen, SM. (eds) Deep Learning: Concepts and Architectures. Studies in Computational Intelligence, vol 866. Springer, Cham. https://doi.org/10.1007/978-3-030-31756-0_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-31756-0_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31755-3
Online ISBN: 978-3-030-31756-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)