Deep Robust Encoder Through Locality Preserving Low-Rank Dictionary

  • Zhengming DingEmail author
  • Ming Shao
  • Yun Fu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9910)


Deep learning has attracted increasing attentions recently due to its appealing performance in various tasks. As a principal way of deep feature learning, deep auto-encoder has been widely discussed in such problems as dimensionality reduction and model pre-training. Conventional auto-encoder and its variants usually involve additive noises (e.g., Gaussian, masking) for training data to learn robust features, which, however, did not consider the already corrupted data. In this paper, we propose a novel Deep Robust Encoder (DRE) through locality preserving low-rank dictionary to extract robust and discriminative features from corrupted data, where a low-rank dictionary and a regularized deep auto-encoder are jointly optimized. First, we propose a novel loss function in the output layer with a learned low-rank clean dictionary and corresponding weights with locality information, which ensures that the reconstruction is noise free. Second, discriminant graph regularizers that preserve the local geometric structure for the data are developed to guide the deep feature learning in each encoding layer. Experimental results on several benchmarks including object and face images verify the effectiveness of our algorithm by comparing with the state-of-the-art approaches.


Auto-encoder Low-rank dictionary Graph regularizer 



This research is supported in part by the NSF CNS award 1314484, ONR award N00014-12-1-1028, ONR Young Investigator Award N00014-14-1-0484, and U.S. Army Research Office Young Investigator Award W911NF-14-1-0218.


  1. 1.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: a deep convolutional activation feature for generic visual recognition. In: International Conference on Machine Learning, pp. 647–655 (2014)Google Scholar
  2. 2.
    Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Neural Information Processing Systems, pp. 2553–2561 (2013)Google Scholar
  3. 3.
    Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708. IEEE (2014)Google Scholar
  4. 4.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  5. 5.
    Bengio, Y.: Learning deep architectures for ai. Found. Trends Mach. Learn. 2(1), 1–127 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Le, Q.V., Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Ng, A.Y.: On optimization methods for deep learning. In: International Conference on Machine Learning, pp. 265–272 (2011)Google Scholar
  7. 7.
    Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: International Conference on Artificial Intelligence and Statistics, pp. 562–570 (2015)Google Scholar
  8. 8.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6791, pp. 44–51. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21735-7_6 CrossRefGoogle Scholar
  10. 10.
    Droniou, A., Sigaud, O.: Gated autoencoders with tied input weights. In: International Conference on Machine Learning, pp. 154–162 (2013)Google Scholar
  11. 11.
    Kan, M., Shan, S., Chen, X.: Bi-shifting auto-encoder for unsupervised domain adaptation. In: IEEE International Conference on Computer Vision, pp. 3846–3854 (2015)Google Scholar
  12. 12.
    Ghifary, M., Bastiaan Kleijn, W., Zhang, M., Balduzzi, D.: Domain generalization for object recognition with multi-task autoencoders. In: IEEE International Conference on Computer Vision, pp. 2551–2559 (2015)Google Scholar
  13. 13.
    Wang, W., Arora, R., Livescu, K., Bilmes, J.: On deep multi-view representation learning. In: International Conference on Machine Learning, pp. 1083–1092 (2015)Google Scholar
  14. 14.
    Xia, C., Qi, F., Shi, G.: Bottom-up visual saliency estimation with deep autoencoder-based sparse reconstruction. IEEE Trans. Neural Netw. Learn. Syst. 27(6), 1227–1240 (2016)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)MathSciNetzbMATHGoogle Scholar
  16. 16.
    Wright, J., Ganesh, A., Rao, S., Peng, Y., Ma, Y.: Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. In: Neural Information Processing Systems, pp. 2080–2088 (2009)Google Scholar
  17. 17.
    Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 171–184 (2013)CrossRefGoogle Scholar
  18. 18.
    Ding, Z., Fu, Y.: Low-rank common subspace for multi-view learning. In: IEEE International Conference on Data Mining, pp. 110–119. IEEE (2014)Google Scholar
  19. 19.
    Shao, M., Kit, D., Fu, Y.: Generalized transfer subspace learning through low-rank constraint. Int. J. Comput. Vis. 109(1–2), 74–93 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Ding, Z., Shao, M., Fu, Y.: Deep low-rank coding for transfer learning. In: Twenty-Fourth International Joint Conference on Artificial Intelligence, pp. 3453–3459 (2015)Google Scholar
  21. 21.
    Jhuo, I.H., Liu, D., Lee, D., Chang, S.F., et al.: Robust visual domain adaptation with low-rank reconstruction. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2175. IEEE (2012)Google Scholar
  22. 22.
    Ma, L., Wang, C., Xiao, B., Zhou, W.: Sparse representation for face recognition based on discriminative low-rank dictionary learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2586–2593. IEEE (2012)Google Scholar
  23. 23.
    Lin, Z., Chen, M., Ma, Y.: The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint (2010). arXiv:1009.5055
  24. 24.
    Cai, J.F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20(4), 1956–1982 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Liu, D.C., Nocedal, J.: On the limited memory bfgs method for large scale optimization. Math. Program. 45(1–3), 503–528 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Li, S., Fu, Y.: Learning robust and discriminative subspace with low-rank constraints. IEEE Trans. Neural Netw. Learn. Syst. PP(99), 1–13 (2015)Google Scholar
  27. 27.
    Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cogn. Neurosci. 3(1), 71–86 (1991)CrossRefGoogle Scholar
  28. 28.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 711–720 (1997)CrossRefGoogle Scholar
  29. 29.
    Liu, G., Yan, S.: Latent low-rank representation for subspace segmentation and feature extraction. In: IEEE International Conference on Computer Vision, pp. 1615–1622 (2011)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringNortheastern UniversityBostonUSA
  2. 2.College of Computer and Information ScienceNortheastern UniversityBostonUSA

Personalised recommendations