Transductive Gaussian Process Regression with Automatic Model Selection

  • Quoc V. Le
  • Alex J. Smola
  • Thomas Gärtner
  • Yasemin Altun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4212)


In contrast to the standard inductive inference setting of predictive machine learning, in real world learning problems often the test instances are already available at training time. Transductive inference tries to improve the predictive accuracy of learning algorithms by making use of the information contained in these test instances. Although this description of transductive inference applies to predictive learning problems in general, most transductive approaches consider the case of classification only. In this paper we introduce a transductive variant of Gaussian process regression with automatic model selection, based on approximate moment matching between training and test data. Empirical results show the feasibility and competitiveness of this approach.


Gaussian Process Test Instance Neural Information Processing System Gaussian Process Regression Moment Match 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Zhu, X.: Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison (2005),
  2. 2.
    Chapelle, O., Zien, A.: Semi-supervised classification by low density separation. In: Tenth International Workshop on Artificial Intelligence and Statistics (2005)Google Scholar
  3. 3.
    Zhu, X., Lafferty, J., Ghahramani, Z.: Semi-supervised learning using gaussian fields and harmonic functions. In: Proc. Intl. Conf. Machine Learning (2003)Google Scholar
  4. 4.
    Smola, A.J., Kondor, I.R.: Kernels and regularization on graphs. In: Schölkopf, B., Warmuth, M.K. (eds.) Proc. Annual Conf.Computational Learning Theory. LNCS, pp. 144–158. Springer, Heidelberg (2003)Google Scholar
  5. 5.
    Pozdnoukhov, A., Bengio, S.: Semi-supervised kernel methods for regression estimation. In: IEEE International Conference on Acoustic, Speech, and Signal Processing (2006)Google Scholar
  6. 6.
    Verbeek, J., Vlassis, N.: Gaussian fields for semi-supervised regression and correspondence learning. Pattern Recognition, special issue on similarity based pattern recognition (2006)Google Scholar
  7. 7.
    Tresp, V.: A Bayesian committee machine. Neural Computation 12(11), 2719–2741 (2000)CrossRefGoogle Scholar
  8. 8.
    Schwaighofer, A., Tresp, V.: Transductive and inductive methods for approximate guassian process regression. In: Neural Information Processing Systems. MIT Press, Cambridge (2003)Google Scholar
  9. 9.
    Chapelle, O., Vapnik, V., Weston, J.: Transductive inference for estimating values of functions. In: Advances in Neural Information Processing Systems (1999)Google Scholar
  10. 10.
    Schuurmans, D., Southey, F., Wilkinson, D., Guo, Y.: Metric-based approaches for semi-supervised regression and classification. In: Semi-Supervised Learning. MIT Press, Cambridge (2006)Google Scholar
  11. 11.
    Brefeld, U., Gärtner, T., Scheffer, T., Wrobel, S.: Efficient co-regularised least squares regression. In: 23rd International Conference on Machine Learning (2006)Google Scholar
  12. 12.
    Gärtner, T., Le, Q., Burton, S., Smola, A.J., Vishwanathan, S.V.N.: Large-scale multiclass transduction. In: Weiss, Y., Schölkopf, B., Platt, J. (eds.) Advances in Neural Information Processing Systems 18, pp. 411–418. MIT Press, Cambridge (2006)Google Scholar
  13. 13.
    Neal, R.M.: Monte Carlo implementation of Gaussian process models for Bayesian regression and classification. Technical Report Technical Report 9702, Dept.of Statistics (1997)Google Scholar
  14. 14.
    Le, Q.V., Smola, A.J., Canu, S.: Heteroscedastic gaussian process regression. In: Proc. Intl. Conf. Machine Learning (2005)Google Scholar
  15. 15.
    Williams, C.K.I.: Prediction with Gaussian processes: From linear regression to linear prediction and beyond. In: Jordan, M.I. (ed.) Learning and Inference in Graphical Models, pp. 599–621. Kluwer Academic, Dordrecht (1998)Google Scholar
  16. 16.
    Neal, R.M.: Assessing relevance determination methods using delve. In: Neural Networks and Machine Learning, pp. 97–129. Springer, Heidelberg (1998)Google Scholar
  17. 17.
    Jordan, M.I., Gharamani, Z., Jaakkola, T.S., Saul, L.K.: An introduction to variational methods for graphical models. In: Jordan, M.I. (ed.) Learning in Graphical Models, pp. 105–162. Kluwer Academic, Dordrecht (1998)Google Scholar
  18. 18.
    Altun, Y., Smola, A.: Divergence minimization and convex duality. In: Proc. Annual Conf. Computational Learning Theory (to appear, 2006)Google Scholar
  19. 19.
    Lütkepohl, H.: Handbook of Matrices. John Wiley and Sons, Chichester (1996)MATHGoogle Scholar
  20. 20.
    Takeuchi, I., Le, Q., Sears, T., Smola, A.: Nonparametric quantile estimation. Journal of Machine Learning Research (to appear, 2006), available at:
  21. 21.
    Newman, D., Hettich, S., Blake, C., Merz, C.: UCI repository of machine learning databases (1998)Google Scholar
  22. 22.
    R Development Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2005) ISBN 3-900051-07-0Google Scholar
  23. 23.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7(1) (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Quoc V. Le
    • 1
  • Alex J. Smola
    • 1
  • Thomas Gärtner
    • 2
  • Yasemin Altun
    • 3
  1. 1.RSISE, Australian National University; Statistical Machine Learning Program, National ICT AustraliaAustralia
  2. 2.Fraunhofer IAISSankt AugustinGermany
  3. 3.Toyota Technological Institute at ChicagoChicagoUSA

Personalised recommendations