Learning Curves for Gaussian Processes via Numerical Cubature Integration

  • Simo Särkkä
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6791)


This paper is concerned with estimation of learning curves for Gaussian process regression with multidimensional numerical integration. We propose an approach where the recursion equations for the generalization error are approximately solved using numerical cubature integration methods. The advantage of the approach is that the eigenfunction expansion of the covariance function does not need to be known. The accuracy of the proposed method is compared to eigenfunction expansion based approximations to the learning curve.


Gaussian process regression learning curve numerical cubature 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    O’Hagan, A.: Curve fitting and optimal design for prediction (with discussion). Journal of the Royal Statistical Society B 40(1), 1–42 (1978)zbMATHGoogle Scholar
  2. 2.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press (2006)Google Scholar
  3. 3.
    Bishop, C.M.: Pattern recognition and machine learning. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  4. 4.
    Opper, M.: Regression with Gaussian processes: Average case performance. In: Theoretical Aspects of Neural Computation. Springer, Heidelberg (1997)Google Scholar
  5. 5.
    Opper, M., Vivarelli, F.: General bounds on bayes errors for regression with gaussian processes. In: NIPS, vol. 11, pp. 302–308. The MIT Press, Cambridge (1999)Google Scholar
  6. 6.
    Sollich, P.: Approximate learning curves for Gaussian processes. In: Proceedings of ICANN 1999, pp. 437–442 (1999)Google Scholar
  7. 7.
    Sollich, P.: Learning curves for Gaussian processes. In: NIPS, vol. 11. The MIT Press, Cambridge (1999)Google Scholar
  8. 8.
    Sollich, P., Halees, A.: Learning curves for Gaussian process regression: Approximations and bounds. Neural Computation 14, 1393–1428 (2002)CrossRefzbMATHGoogle Scholar
  9. 9.
    Williams, C.K.I., Vivarelli, F.: Upper and lower bounds on the learning curve for Gaussian processes. Machine Learning 40, 77–102 (2000)CrossRefGoogle Scholar
  10. 10.
    Sollich, P., Williams, C.: Using the equivalent kernel to understand Gaussian process regression. In: NIPS, vol. 17, pp. 1313–1320. MIT Press, Cambridge (2005)Google Scholar
  11. 11.
    Van Trees, H.L.: Detection, Estimation, and Modulation Theory Part I. John Wiley & Sons, New Yark (1968)zbMATHGoogle Scholar
  12. 12.
    Papoulis, A.: Probability, Random Variables, and Stochastic Processes. McGraw-Hill, New York (1984)zbMATHGoogle Scholar
  13. 13.
    Malzahn, D., Opper, M.: Learning curves for Gaussian process regression: A framework for good approximations. In: NIPS, vol. 13, MIT Press, Cambridge (2001)Google Scholar
  14. 14.
    Malzahn, D., Opper, M.: A variational approach to learning curves. In: NIPS, vol. 14, MIT Press, Cambridge (2002)Google Scholar
  15. 15.
    Ito, K., Xiong, K.: Gaussian filters for nonlinear filtering problems. IEEE Transactions on Automatic Control 45, 910–927 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Wu, Y., Hu, D., Wu, M., Hu, X.: A numerical-integration perspective on Gaussian filters. IEEE Transactions on Signal Processing 54, 2910–2921 (2006)CrossRefGoogle Scholar
  17. 17.
    Arasaratnam, I., Haykin, S.: Cubature Kalman filters. IEEE Transactions on Automatic Control 54, 1254–1269 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Särkkä, S., Hartikainen, J.: On Gaussian optimal smoothing of non-linear state space models. IEEE Transactions on Automatic Control 55, 1938–1941 (2010)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Curtain, R.: A survey of infinite-dimensional filtering. SIAM Review 17(3), 395–411 (1975)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Ray, W.H., Lainiotis, D.G.: Distributed Parameter Systems. Dekker, New York (1978)zbMATHGoogle Scholar
  21. 21.
    Cools, R.: Constructing cubature formulae: The science behind the art. Acta Numerica 6, 1–54 Cambridge University Press, Cambridge (1997)CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Simo Särkkä
    • 1
  1. 1.Department of Biomedical Engineering and Computational ScienceAalto UniversityFinland

Personalised recommendations