Incremental Construction of Low-Dimensional Data Representations

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9896)

Abstract

Various Dimensionality Reduction algorithms transform initial high-dimensional data into their lower-dimensional representations preserving chosen properties of the initial data. Typically, such algorithms use the solution of large-dimensional optimization problems, and the incremental versions are designed for many popular algorithms to reduce their computational complexity. Under manifold assumption about high-dimensional data, advanced manifold learning algorithms should preserve the Data manifold and its differential properties such as tangent spaces, Riemannian tensor, etc. Incremental version of the Grassmann&Stiefel Eigenmaps manifold learning algorithm, which has asymptotically minimal reconstruction error, is proposed in this paper and has significantly smaller computational complexity in contrast to the initial algorithm.

Keywords

Machine learning Dimensionality reduction Manifold learning Tangent bundle manifold learning Incremental learning 

Notes

Acknowledgments

This work is partially supported by the Russian Foundation for Basic Research, research project 16-29-09649 ofi-m.

References

  1. 1.
    Donoho, D.L.: High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality. Lecture at the “Mathematical Challenges of the 21st Century” Conference of the AMS, Los Angeles (2000). http://www-stat.stanford.edu/donoho/Lectures/AMS2000/AMS2000.html
  2. 2.
    Verleysen, M.: Learning high-dimensional data. In: Ablameyko, S., Goras, L., Gori, M., Piuri, V. (eds.) Limitations and Future Trends in Neural Computation. NATO Science Series, III: Computer and Systems Sciences, vol. 186, pp. 141–162. IOS Press, Netherlands (2003)Google Scholar
  3. 3.
    Bengio, Y., Courville, A., Vincent, P.: Representation Learning: A Review and New Perspectives, pp. 1–64 (2014). arXiv:1206.5538v3[cs.LG]. Accessed 23 Apr 2014
  4. 4.
    Bernstein, A., Kuleshov, A.: Low-dimensional data representation in data analysis. In: El Gayar, N., Schwenker, F., Suen, C. (eds.) ANNPR 2014. LNCS, vol. 8774, pp. 47–58. Springer, Heidelberg (2014)Google Scholar
  5. 5.
    Seung, H.S., Lee, D.D.: The manifold ways of perception. Science 290(5500), 2268–2269 (2000)CrossRefGoogle Scholar
  6. 6.
    Huo, X., Ni, X., Smith, A.K.: Survey of manifold-based learning methods. In: Liao, T.W., Triantaphyllou, E. (eds.) Recent Advances in Data Mining of Enterprise Data, pp. 691–745. World Scientific, Singapore (2007)Google Scholar
  7. 7.
    Ma, Y., Fu, Y. (eds.): Manifold Learning Theory and Applications. CRC Press, London (2011)Google Scholar
  8. 8.
    Law, M.H.C., Jain, A.K.: Nonlinear manifold learning for data stream. In: Berry, M., Dayal, U., Kamath, C., Skillicorn, D. (eds.) Proceedings of the 4th SIAM International Conference on Data Mining, Like Buena Vista, Florida, USA, pp. 33–44 (2004)Google Scholar
  9. 9.
    Law, M.H.C., Jain, A.K.: Incremental nonlinear dimensionality reduction by manifold learning. IEEE Trans. Pattern Anal. Mach. Intell. 28(3), 377–391 (2006)CrossRefGoogle Scholar
  10. 10.
    Gao, X., Liang, J.: An improved incremental nonlinear dimensionality reduction for isometric data embedding. Inf. Process. Lett. 115(4), 492–501 (2015)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Saul, L.K., Roweis, S.T.: Think globally, fit locally: unsupervised learning of low dimensional manifolds. J. Mach. Learn. Res. 4, 119–155 (2003)MathSciNetMATHGoogle Scholar
  12. 12.
    Kouropteva, O., Okun, O., Pietikäinen, M.: Incremental locally linear embedding algorithm. In: Kalviainen, H., Parkkinen, J., Kaarna, A. (eds.) SCIA 2005. LNCS, vol. 3540, pp. 521–530. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  13. 13.
    Kouropteva, O., Okun, O., Pietikäinen, M.: Incremental locally linear embedding. Pattern Recogn. 38(10), 1764–1767 (2005)CrossRefMATHGoogle Scholar
  14. 14.
    Schuon, S., Ðurković, M., Diepold, K., Scheuerle, J., Markward, S.: Truly incremental locally linear embedding. In: Proceedings of the CoTeSys 1st International Workshop on Cognition for Technical Systems, 6–8 October 2008, Munich, Germany, p. 5 (2008)Google Scholar
  15. 15.
    Jia, P., Yin, J., Huang, X., Hu, D.: Incremental Laplacian eigenmaps by preserving adjacent information between data points. Pattern Recogn. Lett. 30(16), 1457–1463 (2009)CrossRefGoogle Scholar
  16. 16.
    Liu, X., Yin, J.-w., Feng, Z., Dong, J.: Incremental manifold learning via tangent space alignment. In: Schwenker, F., Marinai, S. (eds.) ANNPR 2006. LNCS (LNAI), vol. 4087, pp. 107–121. Springer, Heidelberg (2006)Google Scholar
  17. 17.
    Abdel-Mannan, O., Ben Hamza, A., Youssef, A.: Incremental line tangent space alignment algorithm. In: Proceedings of 2007 Canadian Conference on Electrical and Computer Engineering (CCECE 2007), 22–26 April 2007, Vancouver, pp. 1329–1332. IEEE (2007)Google Scholar
  18. 18.
    Kuleshov, A., Bernstein, A.: Manifold learning in data mining tasks. In: Perner, P. (ed.) MLDM 2014. LNCS, vol. 8556, pp. 119–133. Springer, Heidelberg (2014)Google Scholar
  19. 19.
    Lee, J.A., Verleysen, M.: Nonlinear Dimensionality Reduction. Information Science and Statistics. Springer, New York (2007)CrossRefMATHGoogle Scholar
  20. 20.
    Lee, J.A., Verleysen, M.: Quality assessment of dimensionality reduction: rank-based criteria. Neurocomputing 72(7–9), 1431–1443 (2009)CrossRefGoogle Scholar
  21. 21.
    Bernstein, A.V., Kuleshov, A.P.: Data-based manifold reconstruction via tangent bundle manifold learning. In: ICML-2014, Topological Methods for Machine Learning Workshop, Beijing, 25 June 2014. http://topology.cs.wisc.edu/KuleshovBernstein.pdf
  22. 22.
    Perrault-Joncas, D., Meilă, M.: Non-linear Dimensionality Reduction: Riemannian Metric Estimation and the Problem of Geometric Recovery, pp. 1–25 (2013). arXiv:1305.7255v1[stat.ML]. Accessed 30 May 2013
  23. 23.
    Jost, J.: Riemannian Geometry and Geometric Analysis, 6th edn. Springer, Berlin (2011)CrossRefMATHGoogle Scholar
  24. 24.
    Bernstein, A.V., Kuleshov, A.P.: Manifold learning: generalizing ability and tangent proximity. Int. J. Softw. Inf. 7(3), 359–390 (2013)Google Scholar
  25. 25.
    Lee, J.M.: Manifolds and Differential Geometry. Graduate Studies in Mathematics, vol. 107. American Mathematical Society, Providence (2009)MATHGoogle Scholar
  26. 26.
    Bernstein, A.V., Kuleshov, A.P.: Tangent bundle manifold learning via Grassmann&Stiefel eigenmaps, pp. 1–25, December 2012. arXiv:1212.6031v1[cs.LG]
  27. 27.
    Jollie, T.: Principal Component Analysis. Springer, New York (2002)Google Scholar
  28. 28.
    Singer, A., Wu, H.-T.: Vector diffusion maps and the connection Laplacian. Commun. Pure Appl. Math. 65(8), 1067–1144 (2012)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Tyagi, H., Vural, E., Frossard, P.: Tangent space estimation for smooth embeddings of Riemannian manifold, pp. 1–35 (2013). arXiv:1208.1065v2[stat.CO]. Accessed 17 May 2013
  30. 30.
    Hamm, J., Daniel, L.D.: Grassmann discriminant analysis: a unifying view on subspace-based learning. In: Proceedings of the 25th International Conference on Machine Learning (ICML 2008), pp. 376–383 (2008)Google Scholar
  31. 31.
    Wolf, L., Shashua, A.: Learning over sets using kernel principal angles. J. Mach. Learn. Res. 4, 913–931 (2003)MathSciNetMATHGoogle Scholar
  32. 32.
    Golub, G.H., Van Loan, C.F.: Matrix Computation, 3rd edn. Johns Hopkins University Press, Baltimore (1996)MATHGoogle Scholar
  33. 33.
    Kuleshov, A., Bernstein, A., Yanovich, Y.: Asymptotically optimal method in manifold estimation. In: Abstracts of the XXIX-th European Meeting of Statisticians, 20–25 July 2013, Budapest, Hungary, p. 325 (2013). http://ems2013.eu/conf/upload/BEK086_006.pdf
  34. 34.
    Genovese, C.R., Perone-Pacifico, M., Verdinelli, I., Wasserman, L.: Minimax manifold estimation. J. Mach. Learn. Res. 13, 1263–1291 (2012)MathSciNetMATHGoogle Scholar
  35. 35.
    Trefethen, L.N.: Bau III, David: Numerical Linear Algebra. SIAM, Philadelphia (1997)CrossRefGoogle Scholar
  36. 36.
    Cormen, T., Leiserson, C., Rivest, R., Stein, C.: Introduction to Algorithms. MIT Press, Cambridge (2001)MATHGoogle Scholar
  37. 37.
    Yanovich, Y.: Asymptotic properties of local sampling on manifolds. J. Math. Stat. (2016)Google Scholar
  38. 38.
    Fredman, M.L., Tarjan, R.E.: Fibonacci heaps and their uses in improved network optimization algorithms. J. Assoc. Comput. Mach. 34(3), 596–615 (1987)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Skolkovo Institute of Science and TechnologyMoscowRussia
  2. 2.Kharkevich Institute for Information Transmission Problems RASMoscowRussia

Personalised recommendations