Approximating Gaussian Processes with \({\cal H}^2\)-Matrices

  • Steffen Börm
  • Jochen Garcke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4701)

Abstract

To compute the exact solution of Gaussian process regression one needs \(\mathcal{O}(N^3)\) computations for direct and \(\mathcal{O}(N^2)\) for iterative methods since it involves a densely populated kernel matrix of size N ×N, here N denotes the number of data. This makes large scale learning problems intractable by standard techniques.

We propose to use an alternative approach: the kernel matrix is replaced by a data-sparse approximation, called an \({\mathcal H}^2\)-matrix. This matrix can be represented by only \({\cal O}(N m)\) units of storage, where m is a parameter controlling the accuracy of the approximation, while the computation of the \({\mathcal H}^2\)-matrix scales with \({\cal O}(N m \log N)\).

Practical experiments demonstrate that our scheme leads to significant reductions in storage requirements and computing times for large data sets in lower dimensional spaces.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)Google Scholar
  2. 2.
    Quiñonero-Candela, J., Rasmussen, C.E.: A unifying view of sparse approximate gaussian process regression. J. of Machine Learning Research 6, 1935–1959 (2005)Google Scholar
  3. 3.
    Shen, Y., Ng, A., Seeger, M.: Fast gaussian process regression using kd-trees. In: Weiss, Y., Schölkopf, B., Platt, J. (eds.) NIPS 18, MIT Press, Cambridge (2006)Google Scholar
  4. 4.
    Freitas, N.D., Wang, Y., Mahdaviani, M., Lang, D.: Fast krylov methods for n-body learning. In: Weiss, Y., Schölkopf, B., Platt, J. (eds.) Advances in Neural Information Processing Systems 18, MIT Press, Cambridge, MA (2006)Google Scholar
  5. 5.
    Lang, D., Klaas, M., de Freitas, N.: Empirical testing of fast kernel density estimation algorithms. Technical Report TR-2005-03, Department of Computer Science, University of British Columbia (2005)Google Scholar
  6. 6.
    Börm, S., Grasedyck, L., Hackbusch, W.: Hierarchical Matrices. Lecture Note 21 of the Max Planck Institute for Mathematics in the Sciences (2003)Google Scholar
  7. 7.
    Grasedyck, L., Hackbusch, W.: Construction and arithmetics of \({\mathcal{H}}\)-matrices. Computing 70(4), 295–334 (2003)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Hackbusch, W., Khoromskij, B., Sauter, S.A.: On \(\mathcal{H}^2\)-matrices. In: Bungartz, H., Hoppe, R., Zenger, C. (eds.) Lect. on Applied Mathematics, pp. 9–29. Springer, Heidelberg (2000)Google Scholar
  9. 9.
    Börm, S., Hackbusch, W.: Data-sparse approximation by adaptive \({\mathcal{H}}^2\)-matrices. Computing 69, 1–35 (2002)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Rifkin, R., Yeo, G., Poggio, T.: Regularized least-squares classification. In: Suykens, J., Horvath, G., Basu, S., Micchelli, C., Vandewalle, J. (eds.) Advances in Learning Theory: Methods, Models and Applications, pp. 131–153. IOS Press, Amsterdam (2003)Google Scholar
  11. 11.
    Greenbaum, A.: Iterative methods for solving linear systems, Philadelphia, PA, USA (1997)Google Scholar
  12. 12.
    Bebendorf, M.: Effiziente numerische Lösung von Randintegralgleichungen unter Verwendung von Niedrigrang-Matrizen. PhD thesis, Uni. Saarbrücken (2000)Google Scholar
  13. 13.
    Börm, S., Grasedyck, L.: HLib – a library for \({\mathcal H}\)- and \({\mathcal H}^2\)-matrices (1999), Available at http://www.hlib.org/
  14. 14.
    Börm, S., Grasedyck, L.: Low-rank approximation of integral operators by interpolation. Computing 72, 325–332 (2004)MATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Grasedyck, L.: Adaptive recompression of \({\mathcal H}\)-matrices for BEM. Computing 74(3), 205–223 (2004)CrossRefMathSciNetGoogle Scholar
  16. 16.
    Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins U. P. (1996)Google Scholar
  17. 17.
    Buonadonna, P., Hellerstein, J., Hong, W., Gay, D., Madden, S.: Task: Sensor network in a box. In: Proc. of European Workshop on Sensor Networks (2005)Google Scholar
  18. 18.
    Ng, A.Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., Liang, E.: Autonomous inverted helicopter flight via reinforcement learning. In: International Symposium on Experimental Robotics (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Steffen Börm
    • 1
  • Jochen Garcke
    • 2
  1. 1.Max Planck Institute for Mathematics in the Sciences, Inselstraße 22–26, 04103 LeipzigGermany
  2. 2.Technische Universität Berlin, Institut für Mathematik, MA 3-3, Straße des 17. Juni 136, 10623 Berlin 

Personalised recommendations