Skip to main content
SpringerLink
Log in

Research of stacked denoising sparse autoencoder

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Learning results depend on the representation of data, so how to efficiently represent data has been a research hot spot in machine learning and artificial intelligence. With the deepening of the deep learning research, studying how to train the deep networks to express high dimensional data efficiently also has been a research frontier. In order to present data more efficiently and study how to express data through deep networks, we propose a novel stacked denoising sparse autoencoder in this paper. Firstly, we construct denoising sparse autoencoder through introducing both corrupting operation and sparsity constraint into traditional autoencoder. Then, we build stacked denoising sparse autoencoders which has multi-hidden layers by layer-wisely stacking denoising sparse autoencoders. Experiments are designed to explore the influences of corrupting operation and sparsity constraint on different datasets, using the networks with various depth and hidden units. The comparative experiments reveal that test accuracy of stacked denoising sparse autoencoder is much higher than other stacked models, no matter what dataset is used and how many layers the model has. We also find that the deeper the network is, the less activated neurons in every layer will have. More importantly, we find that the strengthening of sparsity constraint is to some extent equal to the increase in corrupted level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. MNIST dataset http://yann.lecun.com/exdb/mnist/.

  2. MNIST-Rotation dataset http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/MnistVariations.

  3. STL-10 dataset http://cs.stanford.edu/~acoates/stl10/.

  4. M. Schmidt. minFunc: unconstrained differentiable multivariate optimization in MATLAB. http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html, 2005 http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html.

References

  1. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  2. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  Google Scholar 

  3. Hinton GE (2007) Learning multiple layers of representation. Trends Cogn Sci 11(10):428–434

    Article  Google Scholar 

  4. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127

    Article  Google Scholar 

  5. Hinton GE (2007) To recognize shapes, first learn to generate images. Prog Brain Res 165:535–547

    Article  Google Scholar 

  6. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinovet RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. The computing research repository, 2012.1207(0580)

  7. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11:3371–3408

    MathSciNet  MATH  Google Scholar 

  8. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. ACM, New York, pp 1096–1103

    Google Scholar 

  9. Krizhevsky A, Hinton GE (2010) Using very deep autoencoders for content-based image retrieval. Belgium: i6doc.com publication, pp 489–494

  10. Hinton GE, Krizhevsky A, Wang SD (2011) Transforming auto-encoders. Springer, Espoo, pp 44–51

    Google Scholar 

  11. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  12. Baldi P (2012) Autoencoders, unsupervised learning, and deep architectures. IEEE Transactions on Neural Networks, Bellevue, pp 37–50

    Google Scholar 

  13. Zhang N, Ding S, Shi Z (2016) Denoising Laplacian multi-layer extreme learning machine. Neurocomputing. 171(C):1066–1074.

    Article  Google Scholar 

  14. Zeiler MD, Krishnan D, Taylor GW, Fergus R (2010) Deconvolutional networks. IEEE Computer Society, San Francisco, pp 2528–2535

    Google Scholar 

  15. Zeiler MD, Taylor GW, Fergus R (2011) Adaptive deconvolutional networks for mid and high level feature learning. Piscataway, NJ, USA: IEEE. 2011.2018-25

  16. Serre T, Wolf L, Bileschi S, Riesenhuber M, Poggio T (2007) Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell 29(3):411–426

    Article  Google Scholar 

  17. LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, p 3361

  18. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  19. Jarrett K, Kavukcuoglu K, Ranzato MA, LeCun Y (2009) What is the best multi-stage architecture for object recognition?. IEEE, Kyoto, pp 2146–2153

    Google Scholar 

  20. Le QV, Ranzato AM, Monga R, Devin M, Chen K, Corrado GS, Dean J, Ng AY (2013) Building high level features using large scale unsupervised learning. IEEE, New York, pp 8595–8598

    Google Scholar 

  21. Chandra B, Sharma RK (2014) Adaptive noise schedule for denoising autoencoder. Springer International Publishing, Cham, pp 535–542

    Google Scholar 

  22. Memisevic R (2011) Gradient-based learning of higher-order image features. Barcelona, Spain: 2011 IEEE international conference on computer vision (ICCV), pp 1591–1598

  23. Lemme A, Reinhart RF, Steil JJ (2012) Online learning and generalization of parts-based image representations. Neural Netw 33:194–203

    Article  Google Scholar 

  24. Deng J, Zhang Z, Marchi E, Schuller B (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition. IEEE, New York, pp 511–516

    Google Scholar 

  25. Tan CC, Eswaran C (2011) Using autoencoders for mammogram compression. J Med Syst 35(1):49–58

    Article  Google Scholar 

  26. Lee MK, Han DS (2012) Dimensionality reduction of radio map with nonlinear autoencoder. Electron Lett 48(11):655–657

    Article  Google Scholar 

  27. Liu J, Chi G, Liu Z, Liu Y, Li H, Luo X (2013) Predicting protein structural classes with autoencoder neural networks. IEEE, Piscataway, pp 1894–1899

    Google Scholar 

  28. Gu B, Sheng VS, Tay KY, Romano W, Li S (2015) Incremental Support Vector Learningfor Ordinal Regression. IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 7, pp. 1403–1416

    Article  MathSciNet  Google Scholar 

  29. Palecek K (2014) Extraction of features for lip-reading using autoencoders. Springer, Novi Sad, pp 209–216

    Google Scholar 

  30. Xia Z, Wang X, Sun X, Wang B (2014) Steganalysis of least significant bit matching using multi-order differences. Secur Commun Netw. 7(8):1283–1291

    Article  Google Scholar 

  31. Jiang X, Zhang Y, Zhang W, Xiao X (2013) A novel sparse auto-encoder for deep unsupervised learning. In: 2013 sixth international conference on advanced computational intelligence (ICACI). IEEE Press, Hangzhou, pp 256–261

  32. Le QV, Ngiam J, Coates A, Lahiri A, Prochnow B, Ng AY (2011) On optimization methods for deep learning. Proceedings of the 28th international conference on machine learning (ICML-11), pp 265–272

  33. Xie J, Xu L, Chen E (2012) Image denoising and inpainting with deep neural networks. Nips, pp 341–349

  34. Cho K (2013) Simple sparsification improves sparse denoising autoencoders in denoising highly noisy images. In: Proceedings of the 30th international conference on machine learning (ICML-13), pp 432–440

  35. Chen Y, Hao C, Wu W, Wu E (2016) Robust dense reconstruction by range merging based on confidence estimation. Sci China Inf Sci. 59(9)

  36. Li J-C, Ng WWY, Yeung DS, Chan PPK (2014) Bi-firing deep neural networks. Int J Mach Learn Cybernet 5(1):73–83

    Article  Google Scholar 

  37. Coates A, Lee H, Ng AY (2011) An analysis of single layer networks in unsupervised feature learning. AISTATS, pp 215–223

  38. Makhzani A, Frey B (2014) K-sparse autoencoders. International conference on learning representations, ICLR

  39. Vagin V, Fomina M (2011) Problem of knowledge discovery in noisy databases. Int J Mach Learn Cybernet 2(3):135–145

    Article  Google Scholar 

Download references

Acknowledgements

The work is partially supported by the National Natural Science Foundation of China under Grant No. 61379101 and No.61672522, and the National Basic Research Program of China under Grant No. 2013CB329502.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, China

    Lingheng Meng, Shifei Ding, Nan Zhang & Jian Zhang

  2. Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Lingheng Meng, Shifei Ding, Nan Zhang & Jian Zhang

Authors
  1. Lingheng Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shifei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shifei Ding.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Meng, L., Ding, S., Zhang, N. et al. Research of stacked denoising sparse autoencoder. Neural Comput & Applic 30, 2083–2100 (2018). https://doi.org/10.1007/s00521-016-2790-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-016-2790-x

Keywords

Search

Navigation