Statistical Learning on Manifold-Valued Data

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9729)


Regression on manifolds problem is to estimate an unknown smooth function f that maps p-dimensional manifold-valued inputs, whose values lie on unknown Input manifold M of lower dimensionality q < p embedded in an ambient high-dimensional input space Rp, to m-dimensional outputs from training sample consisting of given ‘input-output’ pairs. We consider this problem in which Jacobian Jf(X) of function f and Input manifold M should be also estimated. The paper presents a new geometrically motivated method for estimating a triple (f(X), Jf(X), M) from given sample. The proposed solution is based on solving a Tangent bundle manifold learning problem for specific unknown Regression manifold embedded in input-output space Rp+m and consisting of input-output pairs (X, f(X)), X ∈ M.


Regression on manifolds Regression on features Input manifold reconstruction Jacobian estimation Tangent bundle manifold learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vapnik, V.: Statistical Learning Theory. John Wiley, New York (1998)MATHGoogle Scholar
  2. 2.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2009)CrossRefMATHGoogle Scholar
  3. 3.
    James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning with Applications in R. Springer Texts in Statistics. Springer, New York (2013)CrossRefMATHGoogle Scholar
  4. 4.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2007)MATHGoogle Scholar
  5. 5.
    Friedman, J.H.: Greedy Function Approximation: A Gradient Boosting Machine. Ann. Stat. 29(5), 1189–1232 (2001)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Rasmussen, C.E., Williams, C.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)MATHGoogle Scholar
  7. 7.
    Loader, C.: Local Regression and Likelihood. Springer, New York (1999)MATHGoogle Scholar
  8. 8.
    Wang, G.: Gary Shan S.: Review of metamodeling techniques in support of engineering design optimization. J. Mech. Des. 129(3), 370–381 (2007)CrossRefGoogle Scholar
  9. 9.
    Forrester, A.I.J., Sobester, A., Keane, A.J.: Engineering Design via Surrogate Modelling: A Practical Guide. Wiley, New York (2008)CrossRefGoogle Scholar
  10. 10.
    Kuleshov, A.P., Bernstein, A.V.: Cognitive technologies in adaptive models of complex plants. Inf. Control Probl. Manuf. 13(1), 1441–1452 (2009)Google Scholar
  11. 11.
    Stone, C.J.: Optimal global rates of convergence for nonparametric regression. Ann. Stat. 10, 1040–1053 (1982)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Seung, H.S., Lee, D.D.: The Manifold Ways of Perception. Science 290(5500), 2268–2269 (2000)CrossRefGoogle Scholar
  13. 13.
    Rajaram, D., Pant, R.S.: An improved methodology for airfoil shape optimization using surrogate based design optimization. In: Rodrigues, H., et al. (eds.) Engineering Optimization IV, pp. 147–152. CRC Press, Taylor & Francis Group, London (2015)Google Scholar
  14. 14.
    Bernstein, A., Kuleshov, A., Sviridenko, Y., Vyshinsky, V.: Fast aerodynamic model for design technology. In: Proceedings of West-East High Speed Flow Field Conference (WEHSFF-2007), Moscow, Russia (2007).
  15. 15.
    Zhu, F., Qin, N., Burnaev, E.V., Bernstein, A.V., Chernova, S.S.: Comparison of three geometric parameterization methods and their effect on aerodynamic optimization. In: Poloni, C. (ed.) Eurogen 2011, Optimization and Control with Applications to Industrial and Societal Problems International Conference on Proceedings - Evolutionary and Deterministic Methods for Design, pp. 758–772. Sira, Capua (2011)Google Scholar
  16. 16.
    Pelletier, B.: Nonparametric regression estimation on closed Riemannian manifolds. J. Nonparametric Stat. 18(1), 57–67 (2006)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Loubes, J.-M., Pelletier, B.: A kernel-based classifier on a Riemannian manifold. Statistics and Decisions 26(1), 35–51 (2008). Verlag, OldenbourgMathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Bickel, P., Li, B.: Local polynomial regression on unknown manifolds. In: Complex Datasets and Inverse Problems: Tomography, Networks and Beyond . IMS Lecture notes – Monograph Series, vol. 54, pp. 177–186 (2007)Google Scholar
  19. 19.
    Aswani, A., Bickel, P., Tomlin, C.: Regression on manifolds: Estimation of the exterior derivative. Ann. Stat. 39(1), 48–81 (2011)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Cheng, M.-Y., Wu, H.-T.: Local Linear Regression on Manifolds and its Geometric Interpretation. J. Am. Stat. Assoc. 108(504), 1421–1434 (2013)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Yang, Y., Dunson, D.B.: Bayesian manifold regression. In: arXiv:1305.0167v2 [math.ST], pp. 1–40, June 16, 2014
  22. 22.
    Guhaniyogi, R., Dunson, D.B.: Compressed gaussian process. In: arXiv:1406.1916v1 [stat.ML], pp. 1–29, June 7, 2014
  23. 23.
    Fletcher, P.T.: Geodesic regression on Riemannian manifolds. In: Proceedings of International Workshop on Mathematical Foundations of Computational Anatomy (MFCA), pp. 75–86 (2011)Google Scholar
  24. 24.
    Hinkle, J., Muralidharan, P., Fletcher, P., Joshi, S.: Polynomial regression on riemannian manifolds. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 1–14. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  25. 25.
    Steinke, F., Hein, M., Schölkopf, B.: Nonparametric regression between general Riemannian manifolds. SIAM J. Imaging Sci. 3(3), 527–563 (2010)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Levina, E., Bickel, P.J.: Maximum likelihood estimation of intrinsic dimension. In: Saul, L., Weiss, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, vol. 17, pp. 777–784. MIT Press, Cambridge (2005)Google Scholar
  27. 27.
    Fan, M., Qiao, H., Zhang, B.: Intrinsic dimension estimation of manifolds by incising balls. Pattern Recogn. 42, 780–787 (2009)CrossRefMATHGoogle Scholar
  28. 28.
    Fan, M., Gu, N., Qiao, H., Zhang, B.: Intrinsic dimension estimation of data by principal component analysis. In: arXiv:1002.2050v1 [cs.CV], pp. 1–8, February 10, 2010
  29. 29.
    Rozza, A., Lombardi, G., Rosa, M., Casiraghi, E., Campadelli, P.: IDEA: intrinsic dimension estimation algorithm. In: Maino, G., Foresti, G.L. (eds.) ICIAP 2011, Part I. LNCS, vol. 6978, pp. 433–442. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  30. 30.
    Ma, Y., Fu, Y. (eds.): Manifold Learning Theory and Applications. CRC Press, London (2011)Google Scholar
  31. 31.
    Bernstein, A.V., Kuleshov, A.P.: Tangent bundle manifold learning via Grassmann&Stiefel eigenmaps. In: arXiv:1212.6031v1 [cs.LG], pp. 1–25, December 2012
  32. 32.
    Bernstein, A.V., Kuleshov, A.P.: Manifold Learning: generalizing ability and tangent proximity. Int. J. Softw. Inf. 7(3), 359–390 (2013)Google Scholar
  33. 33.
    Bernstein, A., Kuleshov, A., Yanovich, Y.: Manifold learning in regression tasks. In: Gammerman, A., Vovk, V., Papadopoulos, H. (eds.) SLDS 2015. LNCS, vol. 9047, pp. 414–423. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  34. 34.
    Niyogi, P., Smale, S., Weinberger, S.: Finding the homology of submanifolds with high confidence from random samples. Discrete Comput. Geom. 39, 419–441 (2008)MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Bernstein, A.V., Kuleshov, A.P.: Data-based manifold reconstruction via tangent bundle manifold learning. In: ICML-2014, Topological Methods for Machine Learning Workshop, Beijing, June 25, 2014.
  36. 36.
    Kuleshov, A., Bernstein, A.: Manifold learning in data mining tasks. In: Perner, P. (ed.) MLDM 2014. LNCS, vol. 8556, pp. 119–133. Springer, Heidelberg (2014)Google Scholar
  37. 37.
    Kuleshov, A., Bernstein, A., Yanovich, Yu.: Asymptotically optimal method in manifold estimation. In: Márkus, L., Prokaj, V. (eds.) Abstracts of the XXIX-th European Meeting of Statisticians, July 20–25, 2013, Budapest, p. 325 (2013)Google Scholar
  38. 38.
    Genovese, C.R., Perone-Pacifico, M., Verdinelli, I., Wasserman, L.: Minimax Manifold Estimation. J. Mach. Learn. Res. 13, 1263–1291 (2012)MathSciNetMATHGoogle Scholar
  39. 39.
    Golub, G.H., Van Loan, C.F.: Matrix Computation, 3rd edn. Johns Hopkins University Press, Baltimore (1996)MATHGoogle Scholar
  40. 40.
    Wasserman, L.: All of Nonparametric Statistics. Springer Texts in Statistics, Berlin (2007)Google Scholar
  41. 41.
    Jollie, T.: Principal Component Analysis. Springer, New York (2002)Google Scholar
  42. 42.
    Singer, A., Wu, H.-T.: Vector Diffusion Maps and the Connection Laplacian. Commun. Pure Appl. Math. 65(8), 1067–1144 (2012)MathSciNetCrossRefMATHGoogle Scholar
  43. 43.
    Tyagi, H., Vural, E., Frossard, P.: Tangent space estimation for smooth embeddings of Riemannian manifold. In: arXiv:1208.1065v2 [stat.CO], pp. 1–35, May 17, 2013
  44. 44.
    Hamm, J., Lee, D.D.: Grassmann discriminant analysis: a unifying view on subspace-based learning. In: Proceedings of the 25th International Conference on Machine Learning (ICML 2008), pp. 376–383 (2008)Google Scholar
  45. 45.
    Wolf, L., Shashua, A.: Learning over sets using kernel principal angles. J. Mach. Learn. Res. 4, 913–931 (2003)MathSciNetMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Skolkovo Institute of Science and TechnologyMoscowRussia
  2. 2.Institute for Systems Analysis, FRC CSC RASMoscowRussia
  3. 3.Kharkevich Institute for Information Transmission Problems RASMoscowRussia

Personalised recommendations