Robust Probabilistic Calibration

  • Stefan Rüping
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4212)


Probabilistic calibration is the task of producing reliable estimates of the conditional class probability P(class | observation) from the outputs of numerical classifiers. A recent comparative study [1] revealed that Isotonic Regression [2] and Platt Calibration [3] are most effective probabilistic calibration technique for a wide range of classifiers. This paper will demonstrate that these methods are sensitive to outliers in the data. An improved calibration method will be introduced that combines probabilistic calibration with methods from the field of robust statistics [4]. It will be shown that the integration of robustness concepts can significantly improve calibration performance.


Calibration Method Scaling Function Decision Function Tonic Regression Isotonic Regression 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Niculescu-Mizil, A., Caruana, R.: Predicting good probabilities with supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 625–632 (2005)Google Scholar
  2. 2.
    Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694–699 (2002)Google Scholar
  3. 3.
    Platt, J.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Smola, A., Bartlett, P., Schölkopf, B., Schuurmans, D. (eds.) Advances in Large Margin Classifiers. MIT Press, Cambridge (1999)Google Scholar
  4. 4.
    Huber, P.J.: Robust Statistics. John Wiley & Sons, Chichester (1981)MATHCrossRefGoogle Scholar
  5. 5.
    Rüping, S.: A simple method for estimating conditional probabilities in SVMs. In: Abecker, A., Bickel, S., Brefeld, U., Drost, I., Henze, N., Herden, O., Minor, M., Scheffer, T., Stojanovic, L., Weibelza hl, S., (eds.) LWA 2004 - Lernen - Wissensentdeckung - Adaptivität, Humboldt-Universität Berlin (2004)Google Scholar
  6. 6.
    Garczarek, U.: Classification Rules in Standardized Partition Spaces. PhD thesis, Universität Dortmund (2002)Google Scholar
  7. 7.
    Rousseeuw, P.J.: Least median of squares regression. J. Am. Stat. Assoc. 79, 871–880 (1984)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Rousseeuw, P.J., Leroy, A.M.: Robust Regression and Outlier Detection. Wiley, Chichester (1987)MATHCrossRefGoogle Scholar
  9. 9.
    Murphy, P.M., Aha, D.W.: UCI repository of machine learning databases (1994)Google Scholar
  10. 10.
    Demsar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)MathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Stefan Rüping
    • 1
  1. 1.Fraunhofer AISSt. AugustinGermany

Personalised recommendations