Similarity-Binning Averaging: A Generalisation of Binning Calibration
In this paper we revisit the problem of classifier calibration, motivated by the issue that existing calibration methods ignore the problem attributes (i.e., they are univariate). We propose a new calibration method inspired in binning-based methods in which the calibrated probabilities are obtained from k instances from a dataset. Bins are constructed by including the k-most similar instances, considering not only estimated probabilities but also the original attributes. This method has been tested wrt. two calibration measures, including a comparison with other traditional calibration methods. The results show that the new method outperforms the most commonly used calibration methods.
KeywordsCalibration Method Calibration Technique Original Attribute Calibration Measure Brier Score
Unable to display preview. Download preview PDF.
- 1.Bella, A., Ferri, C., Hernandez-Orallo, J., Ramirez-Quintana, M.J.: Calibration of machine learning models. In: Handbook of Research on Machine Learning Applications. IGI Global (2009)Google Scholar
- 2.Caruana, R., Niculescu-Mizil, A.: Data mining in metric space: an empirical analysis of supervised learning performance criteria. In: Proc. of the 10th Intl. Conference on Knowledge Discovery and Data Mining, pp. 69–78 (2004)Google Scholar
- 7.Murphy, A.H.: Scalar and vector partitions of the probability score: Part ii. n-state situation. Journal of Applied Meteorology 11, 1182–1192 (1972)Google Scholar
- 8.Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pp. 61–74. MIT Press, Boston (1999)Google Scholar
- 9.Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In: Proc. of the 18th Intl. Conference on Machine Learning, pp. 609–616 (2001)Google Scholar