Abstract
Asymmetric classification problems are characterized by class imbalance or unequal costs for different types of misclassifications. One of the main cited weaknesses of AdaBoost is its perceived inability to handle asymmetric problems. As a result, a multitude of asymmetric versions of AdaBoost have been proposed, mainly as heuristic modifications to the original algorithm. In this paper we challenge this approach and propose instead handling asymmetric tasks by properly calibrating the scores of the original AdaBoost so that they correspond to probability estimates. We then account for the asymmetry using classic decision theoretic approaches. Empirical comparisons of this approach against the most representative asymmetric Adaboost variants show that it compares favorably. Moreover, it retains the theoretical guarantees of the original AdaBoost and it can easily be adjusted to account for changes in class imbalance or costs without need for retraining.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A more intuitive equivalent form is \(\begin{bmatrix} 0&c_{FN} \\ c_{FP}&0 \end{bmatrix}\). Scaling the cost matrix has no effect on the decision problem, so we can divide its entries with \(c_{FP}\), thus assigning a cost of \(1\) to false positives and a cost of \(c=c_{FN}/c_{FP}\) to false negatives.
- 2.
Note that in the binary classification case, a hypothesis \(h_t\) with error \(\epsilon _t > 1/2\) can be turned into one with \(\epsilon _t < 1/2\) simply by flipping its predictions.
- 3.
The mapping of outputs of SVMs to posterior probability estimates exhibits a similar sigmoid distortion to that observed in AdaBoost.
References
Elkan, C.: The foundations of cost-sensitive learning. In: IJCAI (2001)
Fan, W., Stolfo, S.J., Zhang, J., Chan, P.K.: Adacost: misclassification cost-sensitive boosting. In: ICML, pp. 97–105 (1999)
Flach, P.A.: The geometry of roc space: understanding machine learning metrics through roc isometrics. In: AAAI, pp. 194–201 (2003)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comp. Syst. Sci. 55(1), 119–139 (1997)
Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Ann. Stat. 28, 337–407 (2000)
Joshi, M.V., Kumar, V., Agarwal, R.C.: Evaluating boosting algorithms to classify rare classes: comparison and improvements (2001)
Landesa-Vázquez, I., Alba-Castro, J.L.: Shedding light on the asymmetric learning capability of adaboost. Pat. Rec. Lett. 33(3), 247–255 (2012)
Maloof, M.A.: Learning when data sets are imbalanced and when costs are unequal and unknown. In: ICML (2003)
Masnadi-Shirazi, H., Vasconcelos, N.: Asymmetric boosting. In: ICML, pp. 609–619 (2007)
Masnadi-Shirazi, H., Vasconcelos, N.: Cost-sensitive boosting. IEEE Trans. Pat. Anal. Mach. Intell. 33(2), 294–309 (2011)
Niculescu-Mizil, A., Caruana, R.: Obtaining calibrated probabilities from boosting. In: UAI (2005)
Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pp. 61–74. MIT Press (1999)
Robertson, T., Wright, F., Dykstra, R.: Order Restricted Statistical Inference. Probability and Statistics. Wiley, New York (1988)
Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Mach. Learn. 37(3), 297–336 (1999)
Sun, Y., Kamel, M.S., Wong, A.K.C., Wang, Y.: Cost-sensitive boosting for classification of imbalanced data. Pat. Recogn. 40(12), 3358–3378 (2007)
Sun, Y., Wong, A.K.C., Wang, Y.: Parameter inference of cost-sensitive boosting algorithms. In: Perner, P., Imiya, A. (eds.) MLDM 2005. LNCS (LNAI), vol. 3587, pp. 21–30. Springer, Heidelberg (2005)
Ting, K.M.: A comparative study of cost-sensitive boosting algorithms. In: ICML, pp. 983–990 (2000)
Viola, P., Jones, M.: Fast and robust classification using asymmetric adaboost and a detector cascade. In: NIPS (2002)
Wang, Z., Fang, C., Ding, X.: Asymmetric real adaboost. In: ICPR (2008)
Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In: ICML, pp. 609–616 (2001)
Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates (2002)
Acknowledgments
This work was supported by EPSRC grant [EP/I028099/1]. We also thank Peter Flach for suggesting the idea that inspired this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Nikolaou, N., Brown, G. (2015). Calibrating AdaBoost for Asymmetric Learning. In: Schwenker, F., Roli, F., Kittler, J. (eds) Multiple Classifier Systems. MCS 2015. Lecture Notes in Computer Science(), vol 9132. Springer, Cham. https://doi.org/10.1007/978-3-319-20248-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-20248-8_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-20247-1
Online ISBN: 978-3-319-20248-8
eBook Packages: Computer ScienceComputer Science (R0)