Abstract
We propose a boosting algorithm based on AdaBoost for using real-valued weak hypotheses that return confidences of their classifications as real numbers with an approximated upper bound of the training error. The approximated upper bound is induced with Bernoulli’s inequality and the upper bound enables us to analytically calculate a confidence-value that satisfies a reduction in the original upper bound. The experimental results on the Reuters-21578 data set and an Amazon review data show that our boosting algorithm with the perceptron attains better accuracy than Support Vector Machines, decision stumps-based boosting algorithms and a perceptron.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1) (1997)
Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999)
Cohen, W.W., Singer, Y.: A simple, fast, and effective rule learner. In: AAAI 1999/IAAI 1999, pp. 335–342 (1999)
Schapire, R.E., Singer, Y.: Boostexter: A boosting-based system for text categorization. Machine Learning 39(2/3), 135–168 (2000)
Rätsch, G., Onoda, T., Müller, K.R.: Soft margins for adaboost. Machine Learning 42(3), 287–320 (2001)
Pfahringer, B., Holmes, G., Kirkby, R.: Optimizing the induction of alternating decision trees. In: Cheung, D., Williams, G.J., Li, Q. (eds.) PAKDD 2001. LNCS (LNAI), vol. 2035, pp. 477–487. Springer, Heidelberg (2001)
Jin, R., Liu, Y., Si, L., Carbonell, J., Hauptmann, A.G.: A new boosting algorithm using input-dependent regularizer. In: Proc. of ICML 2003 (2003)
Busa-Fekete, R., Kégl, B.: Fast boosting using adversarial bandits. In: Proc. of ICML 2010, pp. 143–150 (2010)
Eaton, E., desJardins, M.: Selective transfer between learning tasks using task-based boosting. In: AAAI (2011)
Quinlan, J.R.: Bagging, boosting, and c4.5. In: AAAI/IAAI, vol. 1, pp. 725–730 (1996)
Collins, M., Schapire, R.E., Singer, Y.: Logistic regression, adaboost and bregman distances. Machine Learning 48(1-3), 253–285 (2002)
Nock, R., Nielsen, F.: A real generalization of discrete adaboost. Artif. Intell. 171(1), 25–41 (2007)
Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain 65(6), 386–408 (1958)
Collins, M.: Discriminative training methods for Hidden Markov Models: theory and experiments with perceptron algorithms. In: Proc. of EMNLP 2002, pp. 1–8 (2002)
Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., Singer, Y.: Online passive-aggressive algorithms. Journal of Machine Learning Research 7, 551–585 (2006)
Escudero, G., Màrquez, L., Rigau, G.: Boosting applied to word sense disambiguation. In: Lopez de Mantaras, R., Plaza, E. (eds.) ECML 2000. LNCS (LNAI), vol. 1810, pp. 129–141. Springer, Heidelberg (2000)
Carreras, X., Màrques, L., Padró, L.: Named entity extraction using adaboost. In: Proc. of CoNLL 2002, pp. 167–170 (2002)
Carreras, X., Màrquez, L., Punyakanok, V., Roth, D.: Learning and inference for clause identification. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 35–47. Springer, Heidelberg (2002)
Schapire, R.E.: Theoretical views of boosting and applications. In: Watanabe, O., Yokomori, T. (eds.) ALT 1999. LNCS (LNAI), vol. 1720, pp. 13–25. Springer, Heidelberg (1999)
Kégl, B., Busa-Fekete, R.: Boosting products of base classifiers. In: Proc. of ICML 2009, pp. 497–504 (2009)
Vapnik, V.: Statistical Learning Theory. John Wiley & Sons (1998)
Blitzer, J., Dredze, M., Pereira, F.: Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In: ACL, pp. 440–447 (2007)
Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. The Annals of Statistics 28(2), 337–407 (2000)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)
Shalev-Shwartz, S., Singer, Y., Srebro, N., Cotter, A.: Pegasos: primal estimated sub-gradient solver for svm. Math. Program. 127(1), 3–30 (2011)
Friedman, J.: Greedy function approximation: A gradient boosting machine. Annals of Statistics 29, 1189–1232 (2000)
Geurts, P., Wehenkel, L., d’Alché Buc, F.: Gradient boosting for kernelized output spaces. In: Proc. of ICML 2007, pp. 289–296 (2007)
Li, L.: Perceptron learning with random coordinate descent (2005)
Abrich, R., Bovbel, P.: CSC2515 fall 2011 boosting with perceptrons and decision stumps (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Iwakura, T., Saitou, T., Okamoto, S. (2014). An AdaBoost for Efficient Use of Confidences of Weak Hypotheses on Text Categorization. In: Pham, DN., Park, SB. (eds) PRICAI 2014: Trends in Artificial Intelligence. PRICAI 2014. Lecture Notes in Computer Science(), vol 8862. Springer, Cham. https://doi.org/10.1007/978-3-319-13560-1_62
Download citation
DOI: https://doi.org/10.1007/978-3-319-13560-1_62
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-13559-5
Online ISBN: 978-3-319-13560-1
eBook Packages: Computer ScienceComputer Science (R0)