Lessons Learned in the Challenge: Making Predictions and Scoring Them

  • Jukka Kohonen
  • Jukka Suomela
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3944)


In this paper we present lessons learned in the Evaluating Predictive Uncertainty Challenge. We describe the methods we used in regression challenges, including our winning method for the Outaouais data set. We then turn our attention to the more general problem of scoring in probabilistic machine learning challenges. It is widely accepted that scoring rules should be proper in the sense that the true generative distribution has the best expected score; we note that while this is useful, it does not guarantee finding the best methods for practical machine learning tasks. We point out some problems in local scoring rules such as the negative logarithm of predictive density (NLPD), and illustrate with examples that many of these problems can be avoided by a distance-sensitive rule such as the continuous ranked probability score (CRPS).


Predictive Distribution Training Point Input Dimension Probabilistic Prediction True Target 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, Heidelberg (2001)CrossRefMATHGoogle Scholar
  2. 2.
    Härdle, W.: Applied Nonparametric Regression. Cambridge University Press, Cambridge (1990)CrossRefMATHGoogle Scholar
  3. 3.
    Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. Technical Report 463, Department of Statistics, University of Washington (2004)Google Scholar
  4. 4.
    Bernardo, J.M., Smith, A.F.M.: Bayesian Theory. John Wiley & Sons, Inc., Chichester (2000)MATHGoogle Scholar
  5. 5.
    Sanders, F.: The verification of probability forecasts. Journal of Applied Meteorology 6, 756–761 (1967)CrossRefGoogle Scholar
  6. 6.
    Smith, C.A.B.: Consistency in statistical inference and decision. Journal of the Royal Statistical Society. Series B 23, 1–37 (1961)MathSciNetMATHGoogle Scholar
  7. 7.
    Savage, L.J.: Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66, 783–801 (1971)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Winkler, R.L.: Probabilistic prediction: Some experimental results. Journal of the American Statistical Association 66, 678–685 (1971)CrossRefGoogle Scholar
  9. 9.
    Corradi, V., Swanson, N.R.: Predictive density evaluation. Technical Report 200419, Rutgers University, Department of Economics (2004)Google Scholar
  10. 10.
    Bremnes, J.B.: Probabilistic forecasts of precipitation in terms of quantiles using NWP model output. Monthly Weather Review 132, 338–347 (2004)CrossRefGoogle Scholar
  11. 11.
    Epstein, E.S.: A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology 8, 985–987 (1969)CrossRefGoogle Scholar
  12. 12.
    Hamill, T.M., Wilks, D.S.: A probabilistic forecast contest and the difficulty in assessing short-range forecast uncertainty. Weather and Forecasting 10, 620–631 (1995)CrossRefGoogle Scholar
  13. 13.
    Matheson, J.E., Winkler, R.L.: Scoring rules for continuous probability distributions. Management Science 22, 1087–1096 (1976)CrossRefMATHGoogle Scholar
  14. 14.
    Bernardo, J.M.: Expected information as expected utility. The Annals of Statistics 7, 686–690 (1979)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Staël von Holstein, C.A.S.: A family of strictly proper scoring rules which are sensitive to distance. Journal of Applied Meteorology 9, 360–364 (1970)CrossRefGoogle Scholar
  16. 16.
    Murphy, A.H.: The ranked probability score and the probability score: A comparison. Monthly Weather Review 98, 917–924 (1970)CrossRefGoogle Scholar
  17. 17.
    Murphy, A.H.: On the “ranked probability score”. Journal of Applied Meteorology 8, 988–989 (1969)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jukka Kohonen
    • 1
  • Jukka Suomela
    • 1
  1. 1.Helsinki Institute for Information Technology, Basic Research Unit, Department of Computer ScienceUniversity of HelsinkiFinland

Personalised recommendations