Machine Learning

, Volume 59, Issue 1–2, pp 161–205 | Cite as

Logistic Model Trees

  • Niels LandwehrEmail author
  • Mark Hall
  • Eibe Frank


Tree induction methods and linear models are popular techniques for supervised learning tasks, both for the prediction of nominal classes and numeric values. For predicting numeric quantities, there has been work on combining these two schemes into ‘model trees’, i.e. trees that contain linear regression functions at the leaves. In this paper, we present an algorithm that adapts this idea for classification problems, using logistic regression instead of linear regression. We use a stagewise fitting process to construct the logistic regression models that can select relevant attributes in the data in a natural way, and show how this approach can be used to build the logistic regression models at the leaves by incrementally refining those constructed at higher levels in the tree. We compare the performance of our algorithm to several other state-of-the-art learning schemes on 36 benchmark UCI datasets, and show that it produces accurate and compact classifiers.


model trees logistic regression classification 


  1. Blake C. and Merz, C. J. (1998). UCI repository of machine learning databases. [].Google Scholar
  2. Breiman, L., Friedman, H., Olshen, J. A., & Stone, C. J. (1984). Classification and Regression Trees. Wadsworth.Google Scholar
  3. Chan, K. Y., & Loh, W. Y. (2004). LOTUS: An algorithm for building accurate and comprehensible logistic regression trees. Journal of Computational and Graphical Statistics, 13:4, 826–852.MathSciNetGoogle Scholar
  4. Frank, E., Wang, Y., Inglis, S., Holmes, G., & Witten, I. H. (1998). Using model trees for classification. Machine Learning, 32:1, 63–76.CrossRefGoogle Scholar
  5. Freund, Y., & Schapire, R. E. (1996). Experiments with a new boosting algorithm. In Proc 13th International Conference on Machine Learning (pp. 148–156). Morgan Kaufmann.Google Scholar
  6. Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: A statistical view of boosting. The Annals of Statistics, 38:2, 337–374.Google Scholar
  7. Gama, J. (2004). Functional trees. Machine Learning, 55:3, 219–250.Google Scholar
  8. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer-Verlag.Google Scholar
  9. Ihaka, R. & Gentleman, R. (1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5:3, 299–314.Google Scholar
  10. Kohavi, R. (1996). Scaling up the accuracy of naive bayes classifiers: A decision-tree hybrid. In Proc 2nd International Conference on Knowledge Discovery and Data Mining (pp. 202–207). Menlo Park, CA: AAAI Press.Google Scholar
  11. Landwehr, N., Hall, M., & Frank, E. (2003). Logistic model trees. In Proc 14th European Conference on Machine Learning (pp. 241–252). Springer-Verlag.Google Scholar
  12. Lim, T.-S., Loh, W. Y., & Shih, Y. S. (2000). A comparison of prediction accuracy, complexity, and training time for thirty-three old and new classification algorithms. Machine Learning, 40:3, 641–666.Google Scholar
  13. Lubinsky, D. (1994). Tree structured interpretable regression. In D. Fisher, and H. J. Lenz (Eds.), Learning from Data (pp. 387–398). Springer-Verlag.Google Scholar
  14. Malerba, D., Appice, A., Ceci, M., & Monopoli, M. (2002). Trading-off local versus global effects of regression nodes in model trees. In M.-S. Hacid, Z. W. Ras, D. A. Zighed, & Y. Kodratoff (Eds.), Proc 13th International Symposium Foundations of Intelligent Systems (pp. 393–402). Springer-Verlag.Google Scholar
  15. Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52:3, 239–281.CrossRefGoogle Scholar
  16. Perlich, C., Provost, F., & Simonoff, J. (2003). Tree inductions vs. logistic regression: A learning-curve analysis. Journal of Machine Learning Research, 4, 211–255.CrossRefGoogle Scholar
  17. Quinlan, J. R. (1992). Learning with continuous classes. In Proc 5th Australian Joint Conference on Artificial Intelligence (pp. 343–348). World Scientific Publishing Company Incorporated.Google Scholar
  18. Quinlan, R. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.Google Scholar
  19. Wang, Y. & Witten, I. (1997). Inducing model trees for continuous classes. In Proc of Poster Papers, European Conf. on Machine Learning. Prague, Czech Republic.Google Scholar
  20. Witten, I. H., & Frank, E. (2000). Data Mining: Practical Machine Learning Tools and Techniques with Java Implemenations. San Francisco: Morgan Kaufmann.Google Scholar

Copyright information

© Springer Science + Business Media, Inc. 2005

Authors and Affiliations

  1. 1.Institute for Computer ScienceUniversity of FreiburgFreiburgGermany
  2. 2.Department of Computer ScienceUniversity of WaikatoHamiltonNew Zealand

Personalised recommendations