Skip to main content
SpringerLink
Log in

Autoencoders

  • Chapter
  • First Online:
Machine Learning for Data Science Handbook

Abstract

An autoencoder is a specific type of a neural network, which is mainly designed to encode the input into a compressed and meaningful representation and then decode it back such that the reconstructed input is similar as possible to the original one. This chapter surveys the different types of autoencoders that are mainly used today. It also describes various applications and use-cases of autoencoders.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 279.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: D. Precup, Y.W. Teh (eds.) Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 70, pp. 214–223. PMLR, International Convention Centre, Sydney, Australia (2017)

    Google Scholar 

  2. Baldi, P.: Autoencoders, unsupervised learning, and deep architectures. In: I. Guyon, G. Dror, V. Lemaire, G. Taylor, D. Silver (eds.) Proceedings of ICML Workshop on Unsupervised and Transfer Learning, Proceedings of Machine Learning Research, vol. 27, pp. 37–49. PMLR, Bellevue, Washington, USA (2012)

    Google Scholar 

  3. Baldi, P., Hornik, K.: Neural networks and principal component analysis: Learning from examples without local minima. Neural Netw. 2(1), 53–58 (1989). https://doi.org/10.1016/0893-6080(89)90014-2

    Article  Google Scholar 

  4. Bank, D., Giryes, R.: An ETF view of dropout regularization. In: 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7–10, 2020. BMVA Press (2020). https://www.bmvc2020-conference.com/assets/papers/0044.pdf

  5. Belghazi, M.I., Rajeswar, S., Mastropietro, O., Rostamzadeh, N., Mitrovic, J., Courville, A.: Hierarchical adversarially learned inference (2018)

    Google Scholar 

  6. Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg (2006)

    MATH  Google Scholar 

  7. Bousquet, O., Gelly, S., Tolstikhin, I., Simon-Gabriel, C.J., Schoelkopf, B.: From optimal transport to generative modeling: the vegan cookbook. arXiv (2017)

    Google Scholar 

  8. Burda, Y., Grosse, R.B., Salakhutdinov, R.: Importance weighted autoencoders. CoRR abs/1509.00519 (2015)

    Google Scholar 

  9. Chen, M., Xu, Z., Weinberger, K., Sha, F.: Marginalized denoising autoencoders for domain adaptation. Proceedings of the 29th International Conference on Machine Learning, ICML 2012 1 (2012)

    Google Scholar 

  10. Dilokthanakul, N., Mediano, P.A.M., Garnelo, M., Lee, M.C.H., Salimbeni, H., Arulkumaran, K., Shanahan, M.: Deep unsupervised clustering with gaussian mixture variational autoencoders. ArXiv abs/1611.02648 (2017)

    Google Scholar 

  11. Duda, R.O., Hart, P.E., Stork, D.G., et al.: Pattern classification. International Journal of Computational Intelligence and Applications 1, 335–339 (2001)

    MATH  Google Scholar 

  12. Dumoulin, V., Belghazi, I., Poole, B., Lamb, A., Arjovsky, M., Mastropietro, O., Courville, A.C.: Adversarially learned inference. ArXiv abs/1606.00704 (2016)

    Google Scholar 

  13. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11, 625–660 (2010)

    MathSciNet  MATH  Google Scholar 

  14. Feigin, Y., Spitzer, H., Giryes, R.: GMM-based generative adversarial encoder learning (2020)

    Google Scholar 

  15. Friedman, J.H.: On bias, variance, 0/1—loss, and the curse-of-dimensionality. Data mining and knowledge discovery 1(1), 55–77 (1997)

    Article  Google Scholar 

  16. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 2414–2423 (2016)

    Google Scholar 

  17. Gogoi, M., Begum, S.A.: Image classification using deep autoencoders. In: 2017 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), pp. 1–5 (2017). https://doi.org/10.1109/ICCIC.2017.8524276

  18. Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., van den Hengel, A.: Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection (2019)

    Google Scholar 

  19. Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. J. Mach. Learn. Res. 13(null), 723–773 (2012)

    Google Scholar 

  20. Guo, X., Liu, X., Zhu, E., Yin, J.: Deep clustering with convolutional autoencoders. pp. 373–382 (2017)

    Google Scholar 

  21. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S.: Learning temporal regularity in video sequences. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 733–742 (2016)

    Google Scholar 

  22. Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining Collaborative Filtering Recommendations. In: Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, CSCW ’00, pp. 241–250. ACM (2000)

    Google Scholar 

  23. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M.M., Mohamed, S., Lerchner, A.: beta-VAE: Learning basic visual concepts with a constrained variational framework. In: ICLR (2017)

    Google Scholar 

  24. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. science 313(5786), 504–507 (2006)

    Google Scholar 

  25. Hochberg, D.C., Giryes, R., Greenspan, H.: A self supervised Stylegan for Classification with extremely limited annotations (2021)

    Google Scholar 

  26. Hou, X., Shen, L., Sun, K., Qiu, G.: Deep feature consistent variational autoencoder. CoRR abs/1610.00291 (2016). http://arxiv.org/abs/1610.00291

  27. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. CoRR abs/1312.6114 (2013)

    Google Scholar 

  28. Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O.: Autoencoding beyond pixels using a learned similarity metric. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning—Volume 48, ICML’16, p. 1558–1566. JMLR.org (2016)

    Google Scholar 

  29. Le, L., Patterson, A., White, M.: Supervised autoencoders: Improving generalization performance with unsupervised regularizers. In: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (eds.) Advances in Neural Information Processing Systems 31, pp. 107–117. Curran Associates, Inc. (2018)

    Google Scholar 

  30. Li, C., Liu, H., Chen, C., Pu, Y., Chen, L., Henao, R., Carin, L.: Alice: Towards understanding adversarial learning for joint distribution matching. In: I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/ade55409d1224074754035a5a937d2e0-Paper.pdf

  31. Liang, D., Krishnana, R.G., Hoffman, M.D., Jebara, T.: Variational autoencoders for collaborative filtering. CoRR abs/1802.05814 (2018). https://arxiv.org/abs/1802.05814

  32. Lin, J.: Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 37(1), 145–151 (1991). https://doi.org/10.1109/18.61115. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=61115

  33. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, p. 3730–3738. IEEE Computer Society, USA (2015). https://doi.org/10.1109/ICCV.2015.425

  34. Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I.J.: Adversarial autoencoders. CoRR abs/1511.05644 (2015)

    Google Scholar 

  35. Masci, J., Meier, U., Cireşan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: T. Honkela, W. Duch, M. Girolami, S. Kaski (eds.) Artificial Neural Networks and Machine Learning—ICANN 2011, pp. 52–59. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)

    Chapter  Google Scholar 

  36. van den Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks (2016)

    Google Scholar 

  37. Oord, A.v.d., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K.: Conditional image generation with pixelCNN decoders. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pp. 4797–4805. Curran Associates Inc., USA (2016). http://dl.acm.org/citation.cfm?id=3157382.3157633

  38. Plaut, E.: From principal subspaces to principal components with linear autoencoders (2018)

    Google Scholar 

  39. Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., Carin, L.: Variational autoencoder for deep learning of images, labels and captions. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2352–2360 (2016)

    Google Scholar 

  40. Ranzato, M., Huang, F.J., Boureau, Y., LeCun, Y.: Unsupervised learning of invariant feature hierarchies with applications to object recognition. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007). https://doi.org/10.1109/CVPR.2007.383157

  41. Ricci, F., Rokach, L., Shapira, B.: Introduction to recommender systems handbook. In: Recommender systems handbook, pp. 1–35. Springer (2011)

    Google Scholar 

  42. Rubner, Y., Tomasi, C., Guibas, L.: The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision 40, 99–121 (2000). https://doi.org/10.1023/A:1026543900054

    Article  MATH  Google Scholar 

  43. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1. chap. Learning Internal Representations by Error Propagation, pp. 318–362. MIT Press, Cambridge, MA, USA (1986). http://dl.acm.org/citation.cfm?id=104279.104293

  44. Sedhain, S., Menon, A.K., Sanner, S., Xie, L.: AutoRec: Autoencoders meet collaborative filtering. In: Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18–22, 2015—Companion Volume, pp. 111–112 (2015)

    Google Scholar 

  45. Song, C., Liu, F., Huang, Y., Wang, L., Tan, T.: Auto-encoder based data clustering. In: J. Ruiz-Shulcloper, G. Sanniti di Baja (eds.) Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 117–124. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)

    Chapter  Google Scholar 

  46. Strub, F., Mary, J.: Collaborative Filtering with Stacked Denoising AutoEncoders and Sparse Inputs. In: NIPS Workshop on Machine Learning for eCommerce. Montreal, Canada (2015)

    Google Scholar 

  47. Strub, F., Mary, J., Gaudel, R.: Hybrid recommender system based on autoencoders. CoRR abs/1606.07659 (2016). http://arxiv.org/abs/1606.07659

  48. Tenenbaum, J.B., De Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. science 290(5500), 2319–2323 (2000)

    Google Scholar 

  49. Tolstikhin, I., Bousquet, O., Gelly, S., Scholkopf, B.: Wasserstein auto-encoders. ICML 2018 (2018)

    Google Scholar 

  50. Van Den Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning—Volume 48, ICML’16, p. 1747–1756. JMLR.org (2016)

    Google Scholar 

  51. Van Der Maaten, L., Postma, E., Van den Herik, J.: Dimensionality reduction: a comparative review. J Mach Learn Res 10, 66–71 (2009)

    Google Scholar 

  52. Villani, C.: Topics in Optimal Transportation. Graduate studies in mathematics. American Mathematical Society (2003). https://books.google.co.il/books?id=GqRXYFxe0l0C

    Book  MATH  Google Scholar 

  53. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pp. 1096–1103. ACM, New York, NY, USA (2008). https://doi.org/10.1145/1390156.1390294. http://doi.acm.org/10.1145/1390156.1390294

  54. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010). http://dl.acm.org/citation.cfm?id=1756006.1953039

    MathSciNet  MATH  Google Scholar 

  55. Wagstaff, K., Cardie, C., Rogers, S., Schrödl, S.: Constrained k-means clustering with background knowledge. In: Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, p. 577–584. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (2001)

    Google Scholar 

  56. Wang, W., Huang, Y., Wang, Y., Wang, L.: Generalized autoencoder: A neural network framework for dimensionality reduction. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 490–497 (2014)

    Google Scholar 

  57. Wang, Y., Yao, H., Zhao, S.: Auto-encoder based dimensionality reduction. Neurocomputing 184, 232–242 (2016)

    Article  Google Scholar 

  58. Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemometrics and intelligent laboratory systems 2(1–3), 37–52 (1987)

    Article  Google Scholar 

  59. Wu, Y., DuBois, C., Zheng, A.X., Ester, M.: Collaborative denoising auto-encoders for top-n recommender systems. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, San Francisco, CA, USA, February 22–25, 2016, pp. 153–162 (2018)

    Google Scholar 

  60. Zhang, Y., Lee, K., Lee, H.: Augmenting supervised neural networks with unsupervised objectives for large-scale image classification. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning—Volume 48, ICML’16, p. 612–621. JMLR.org (2016)

    Google Scholar 

  61. Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., Hua, X.S.: Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, p. 1933–1941. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3123266.3123451

  62. Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., ki Cho, D., Chen, H.: Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: ICLR (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Tel Aviv University, Tel Aviv, Israel

    Dor Bank & Raja Giryes

  2. Department of Industrial Engineering, Faculty of Engineering, Tel Aviv University, Tel Aviv, Israel

    Noam Koenigstein

Authors
  1. Dor Bank
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Noam Koenigstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raja Giryes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Noam Koenigstein .

Editor information

Editors and Affiliations

  1. Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel

    Lior Rokach

  2. Department of Industrial Engineering, Tel Aviv University, Ramat Aviv, Israel

    Oded Maimon

  3. Department of Industrial Engineering, Tel Aviv University, Tel Aviv, Israel

    Erez Shmueli

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Bank, D., Koenigstein, N., Giryes, R. (2023). Autoencoders. In: Rokach, L., Maimon, O., Shmueli, E. (eds) Machine Learning for Data Science Handbook. Springer, Cham. https://doi.org/10.1007/978-3-031-24628-9_16

Download citation

Publish with us

Policies and ethics

Search

Navigation