Abstract
The statistical learning theory has formulated the Structural Risk Minimization (SRM) principle, based upon the functional form of risk bound on the generalization performance of a learning machine. This paper addresses the application of this formula, which is equivalent to a complexity penalty, to model selection tasks for decision trees, whereas the quantization of the machine capacity for decision trees is estimated using an empirical approach. Experimental results show that, for either classification or regression problems, this novel strategy of decision tree pruning performs better than alternative methods. We name classification and regression trees pruned by virtue of this methodology as Statistical Learning Intelligent Trees (SLIT).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)
Cherkassky, V., Shao, X., Mulier, F., Vapnik, V.: Model Complexity Control for Regression Using VC Generalization Bounds. IEEE Trans. Neural Networks 10, 1075–1089 (1999)
Vapnik, V., Levin, E., LeCun, Y.: Measuring the VC-dimension of a Learning Machine. Neural Computation 6, 851–876 (1994)
Shao, X., Cherkassky, V., Li, W.: Measuring the VC-dimension Using Optimized Experimental Design. Neural Computation 12, 1969–1986 (2000)
Yang, Z., Ji, L.: A New Way to Estimate the VC-dimension with Application to Decision Trees (Submitted). Technical report, DA-050812, Inst. of Information Processing, Dept. of Automation, Tsinghua University (2005)
Vapnik, V.: Estimation of Dependences Based on Empirical Data. Springer, Heidelberg (1982)
Cherkassky, V., Ma, Y.Q.: Comparison of Model Selection for Regression. Neural Computation 15, 1691–1714 (2003)
Cortes, C., Vapnik, V.: Support-Vector Networks. Machine Learning 20, 273–297 (1995)
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Chapman and Hall, Boca Raton (1993)
Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)
Mansour, Y.: Pessimistic Decision Tree Pruning Based on Tree Size. In: Proc. 14th Intl’ Conf. on Machine Learning – ICML 1997, pp. 195–201 (1997)
Blake, C.L., Merz, C.J.: UCI Repository of Machine Learning Databases. Dept. of Information and Computer Science. University of California, Irvine (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yang, Z., Zhu, W., Ji, L. (2006). SLIT: Designing Complexity Penalty for Classification and Regression Trees Using the SRM Principle. In: Wang, J., Yi, Z., Zurada, J.M., Lu, BL., Yin, H. (eds) Advances in Neural Networks - ISNN 2006. ISNN 2006. Lecture Notes in Computer Science, vol 3971. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11759966_131
Download citation
DOI: https://doi.org/10.1007/11759966_131
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34439-1
Online ISBN: 978-3-540-34440-7
eBook Packages: Computer ScienceComputer Science (R0)