SLIT: Designing Complexity Penalty for Classification and Regression Trees Using the SRM Principle

  • Zhou Yang
  • Wenjie Zhu
  • Liang Ji
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3971)


The statistical learning theory has formulated the Structural Risk Minimization (SRM) principle, based upon the functional form of risk bound on the generalization performance of a learning machine. This paper addresses the application of this formula, which is equivalent to a complexity penalty, to model selection tasks for decision trees, whereas the quantization of the machine capacity for decision trees is estimated using an empirical approach. Experimental results show that, for either classification or regression problems, this novel strategy of decision tree pruning performs better than alternative methods. We name classification and regression trees pruned by virtue of this methodology as Statistical Learning Intelligent Trees (SLIT).


Regression Tree Empirical Risk Statistical Learn Theory Tree Pruning Pruning Strategy 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)MATHGoogle Scholar
  2. 2.
    Cherkassky, V., Shao, X., Mulier, F., Vapnik, V.: Model Complexity Control for Regression Using VC Generalization Bounds. IEEE Trans. Neural Networks 10, 1075–1089 (1999)CrossRefGoogle Scholar
  3. 3.
    Vapnik, V., Levin, E., LeCun, Y.: Measuring the VC-dimension of a Learning Machine. Neural Computation 6, 851–876 (1994)CrossRefGoogle Scholar
  4. 4.
    Shao, X., Cherkassky, V., Li, W.: Measuring the VC-dimension Using Optimized Experimental Design. Neural Computation 12, 1969–1986 (2000)CrossRefGoogle Scholar
  5. 5.
    Yang, Z., Ji, L.: A New Way to Estimate the VC-dimension with Application to Decision Trees (Submitted). Technical report, DA-050812, Inst. of Information Processing, Dept. of Automation, Tsinghua University (2005)Google Scholar
  6. 6.
    Vapnik, V.: Estimation of Dependences Based on Empirical Data. Springer, Heidelberg (1982)MATHGoogle Scholar
  7. 7.
    Cherkassky, V., Ma, Y.Q.: Comparison of Model Selection for Regression. Neural Computation 15, 1691–1714 (2003)MATHCrossRefGoogle Scholar
  8. 8.
    Cortes, C., Vapnik, V.: Support-Vector Networks. Machine Learning 20, 273–297 (1995)MATHGoogle Scholar
  9. 9.
    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Chapman and Hall, Boca Raton (1993)Google Scholar
  10. 10.
    Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)MATHGoogle Scholar
  11. 11.
    Mansour, Y.: Pessimistic Decision Tree Pruning Based on Tree Size. In: Proc. 14th Intl’ Conf. on Machine Learning – ICML 1997, pp. 195–201 (1997)Google Scholar
  12. 12.
    Blake, C.L., Merz, C.J.: UCI Repository of Machine Learning Databases. Dept. of Information and Computer Science. University of California, Irvine (1998), Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Zhou Yang
    • 1
  • Wenjie Zhu
    • 2
  • Liang Ji
    • 1
  1. 1.State Key Laboratory of Intelligent Technology and Systems & Institute of Information Processing, Dept. of AutomationTsinghua UniversityBeijingChina
  2. 2.Dept. of Statistics and Actuarial SciencesThe University of Hong KongHong Kong S.A.R.

Personalised recommendations