Evaluating Predictive Uncertainty Challenge

  • Joaquin Quiñonero-Candela
  • Carl Edward Rasmussen
  • Fabian Sinz
  • Olivier Bousquet
  • Bernhard Schölkopf
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3944)

Abstract

This Chapter presents the PASCAL Evaluating Predictive Uncertainty Challenge, introduces the contributed Chapters by the participants who obtained outstanding results, and provides a discussion with some lessons to be learnt. The Challenge was set up to evaluate the ability of Machine Learning algorithms to provide good “probabilistic predictions”, rather than just the usual “point predictions” with no measure of uncertainty, in regression and classification problems. Parti-cipants had to compete on a number of regression and classification tasks, and were evaluated by both traditional losses that only take into account point predictions and losses we proposed that evaluate the quality of the probabilistic predictions.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Rasmussen, C.E., Quiñonero-Candela, J.: Healing the relevance vector machine by augmentation. In: De Raet, Wrobel (eds.) Proceedings of the 22nd International Conference on Machine Learning, pp. 689–696. ACM Press, New York (2005)Google Scholar
  2. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996)MATHGoogle Scholar
  3. Guyon, I., Gunn, S., Ben-Hur, A., Dror, G.: Result analysis of the nips 2003 feature selection challenge. In: Saul, L.K., Weiss, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, Cambridge, Massachussetts, vol. 17, pp. 545–552. The MIT Press, Cambridge (2005)Google Scholar
  4. Hanley, J.A., McNeil, B.J.: The meaning and use ofthe Area under a Receiver Operating Characteristic ROC Curve. Radiology 143(1), 26–29 (1982)CrossRefGoogle Scholar
  5. Sinz, F.H., Candela, J.Q., Bakır, G.H., Rasmussen, C.E., Franz, M.O.: Learning depth from stereo. In: Rasmussen, C.E., Bülthoff, H.H., Schölkopf, B., Giese, M.A. (eds.) DAGM 2004. LNCS, vol. 3175, pp. 245–252. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Smola, A.J., Bartlett, P., Schölkopf, B., Schuurmans, D. (eds.) Advances in Large Margin Classifiers. MIT Press, Cambridge (1999)Google Scholar
  7. Langford, J., Zadrozny, B.: Estimating class membership probabilities using classifier learners. In: Cowell, R.G., Ghahramani, Z. (eds.) Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics. Society for Artificial Intelligence and Statistics, pp. 198–205 (2005), (Available electronically at http://www.gatsby.ucl.ac.uk/aistats/)
  8. Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Joaquin Quiñonero-Candela
    • 1
    • 2
    • 3
  • Carl Edward Rasmussen
    • 1
  • Fabian Sinz
    • 1
  • Olivier Bousquet
    • 1
    • 4
  • Bernhard Schölkopf
    • 1
  1. 1.Max Planck Institute for Biological CyberneticsTübingenGermany
  2. 2.Fraunhofer FIRST.IDABerlinGermany
  3. 3.TU Berlin, SWTBerlinGermany
  4. 4.PertinenceParisFrance

Personalised recommendations