Manifold Learning in Regression Tasks

  • Alexander Bernstein
  • Alexander Kuleshov
  • Yury YanovichEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9047)


The paper presents a new geometrically motivated method for non-linear regression based on Manifold learning technique. The regression problem is to construct a predictive function which estimates an unknown smooth mapping f from q-dimensional inputs to m-dimensional outputs based on a training data set consisting of given ‘input-output’ pairs. The unknown mapping f determines q-dimensional manifold M(f) consisting of all the ‘input-output’ vectors which is embedded in (q+m)-dimensional space and covered by a single chart; the training data set determines a sample from this manifold. Modern Manifold Learning methods allow constructing the certain estimator M* from the manifold-valued sample which accurately approximates the manifold. The proposed method called Manifold Learning Regression (MLR) finds the predictive function fMLR to ensure an equality M(fMLR) = M*. The MLR simultaneously estimates the m×q Jacobian matrix of the mapping f.


Nonlinear regression Dimensionality reduction Manifold learning Tangent bundle manifold learning Manifold learning regression 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vapnik, V.: Statistical Learning Theory. John Wiley, New-York (1998)zbMATHGoogle Scholar
  2. 2.
    James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning with Applications in R. Springer Texts in Statistics, New-YorkGoogle Scholar
  3. 3.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer (2009)Google Scholar
  4. 4.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2007)Google Scholar
  5. 5.
    Deng, L., Yu, D.: Deep Learning: Methods and Applications. NOW Publishers, Boston (2014)Google Scholar
  6. 6.
    Breiman, L.: Random Forests. Machine Learning 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  7. 7.
    Friedman, J.H.: Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29(5), 1189–1232 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Rasmussen, C.E., Williams, C.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  9. 9.
    Belyaev, M., Burnaev, E., Kapushev, Y.: Gaussian process regression for structured data sets. To appear in Proceedings of the SLDS 2015, London, England, UK (2015)Google Scholar
  10. 10.
    Burnaev E., Panov M.: Adaptive design of experiments based on gaussian processes. To appear in Proceedings of the SLDS 2015, London, England, UK (2015)Google Scholar
  11. 11.
    Loader, C.: Local Regression and Likelihood. Springer, New York (1999)zbMATHGoogle Scholar
  12. 12.
    Vejdemo-Johansson, M.: Persistent homology and the structure of data. In: Topological Methods for Machine Learning, an ICML 2014 Workshop, Beijing, China, June 25 (2014).
  13. 13.
    Carlsson, G.: Topology and Data. Bull. Amer. Math. Soc. 46, 255–308 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Edelsbrunner, H., Harer, J.: Computational Topology: An Introduction. Amer. Mathematical Society (2010)Google Scholar
  15. 15.
    Cayton, L.: Algorithms for manifold learning. Univ of California at San Diego (UCSD), Technical Report CS2008-0923, pp. 541-555. Citeseer (2005)Google Scholar
  16. 16.
    Huo, X., Ni, X., Smith, A.K.: Survey of manifold-based learning methods. In: Liao, T.W., Triantaphyllou, E. (eds.) Recent Advances in Data Mining of Enterprise Data, pp. 691–745. World Scientific, Singapore (2007)Google Scholar
  17. 17.
    Ma, Y., Fu, Y. (eds.): Manifold Learning Theory and Applications. CRC Press, London (2011)Google Scholar
  18. 18.
    Bernstein, A.V., Kuleshov, A.P.: Tangent bundle manifold learning via grassmann&stiefel eigenmaps. In: arXiv:1212.6031v1 [cs.LG], pp. 1-25, December 2012Google Scholar
  19. 19.
    Bernstein, A.V., Kuleshov, A.P.: Manifold Learning: generalizing ability and tangent proximity. International Journal of Software and Informatics 7(3), 359–390 (2013)Google Scholar
  20. 20.
    Kuleshov, A., Bernstein, A.: Manifold learning in data mining tasks. In: Perner, P. (ed.) MLDM 2014. LNCS, vol. 8556, pp. 119–133. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  21. 21.
    Kuleshov, A., Bernstein, A., Yanovich, Yu.: Asymptotically optimal method in Manifold estimation. In: Márkus, L., Prokaj, V. (eds.) Abstracts of the XXIX-th European Meeting of Statisticians, July 20-25, Budapest, p. 325 (2013)Google Scholar
  22. 22.
    Genovese, C.R., Perone-Pacifico, M., Verdinelli, I., Wasserman, L.: Minimax Manifold Estimation. Journal Machine Learning Research 13, 1263–1291 (2012)zbMATHMathSciNetGoogle Scholar
  23. 23.
    Kuleshov, A.P., Bernstein, A.V.: Cognitive Technologies in Adaptive Models of Complex Plants. Information Control Problems in Manufacturing 13(1), 1441–1452 (2009)Google Scholar
  24. 24.
    Bunte, K., Biehl, M., Hammer B.: Dimensionality reduction mappings. In: Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2011), pp. 349-356. IEEE, Paris (2011)Google Scholar
  25. 25.
    Lee, J.A.: Verleysen, M.: Quality assessment of dimensionality reduction: Rank-based criteria. Neurocomputing 72(7–9), 1431–1443 (2009)CrossRefGoogle Scholar
  26. 26.
    Saul, L.K., Roweis, S.T.: Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research 4, 119–155 (2003)MathSciNetGoogle Scholar
  27. 27.
    Saul, L.K., Roweis, S.T.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  28. 28.
    Zhang, Z., Zha, H.: Principal Manifolds and Nonlinear Dimension Reduction via Local Tangent Space Alignment. SIAM Journal on Scientific Computing 26(1), 313–338 (2005)CrossRefMathSciNetGoogle Scholar
  29. 29.
    Hamm, J., Lee, D.D.: Grassmann discriminant analysis: A unifying view on subspace-based learning. In: Proceedings of the 25th International Conference on Machine Learning (ICML 2008), pp. 376-83 (2008)Google Scholar
  30. 30.
    Tyagi, H., Vural, E., Frossard, P.: Tangent space estimation for smooth embeddings of riemannian manifold. In: arXiv:1208.1065v2 [stat.CO], pp. 1-35, May 17 (2013)Google Scholar
  31. 31.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15, 1373–1396 (2003)CrossRefzbMATHGoogle Scholar
  32. 32.
    Bengio, Y., Monperrus, M.: Non-local manifold tangent learning. In: Advances in Neural Information Processing Systems, vol. 17, pp. 129-136. MIT Press, Cambridge (2005)Google Scholar
  33. 33.
    Dollár, P., Rabaud, V., Belongie, S.: Learning to traverse image manifolds. In: Advances in Neural Information Processing Systems, vol. 19, pp. 361-368. MIT Press, Cambridge (2007)Google Scholar
  34. 34.
    Xiong, Y., Chen, W., Apley, D., Ding, X.: A Nonstationary Covariance-Based Kriging Method for Metamodeling in Engineering Design. International Journal for Numerical Methods in Engineering 71(6), 733–756 (2007)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Alexander Bernstein
    • 1
    • 2
  • Alexander Kuleshov
    • 1
    • 2
  • Yury Yanovich
    • 1
    • 2
    Email author
  1. 1.Kharkevich Institute for Information Transmission Problems RASMoscowRussia
  2. 2.National Research University Higher School of EconomicsMoscowRussia

Personalised recommendations