Modeling Threshold Interaction Effects Through the Logistic Classification Trunk

Abstract

We introduce a model dealing with the identification of interaction effects in binary response data, which integrates recursive partitioning and generalized linear models. It derives from an ad-hoc specification and consequent implementation of the Simultaneous Threshold Interaction Modeling Algorithm (STIMA). The model, called Logistic Classification Trunk, allows us to obtain regression parameters by maximum likelihood through the simultaneous estimation of both main effects and threshold interaction effects. The main feature of this model is that it allows the user to evaluate a unique model and simultaneously the importance of both effects obtained by first growing a classification trunk and then by pruning it back to avoid overfitting. We investigate the choice of a suitable pruning parameter through a simulation study and compare the classification accuracy of the Logistic Classification Trunk with that of 13 alternative models/classifiers on 25 binary response datasets.

This is a preview of subscription content, log in to check access.

References

  1. ALLEN, J., and LE, H. (2008), “An Additional Measure of Overall Effect Size for Logistic Regression Models”, Journal of Educational and Behavioral Statistics, 33(4), 416–441.

    Article  Google Scholar 

  2. ASSMANN, S.F., HOSMER, D.W., LEMESHOW, S., and MUNDT, K.A. (1996), “Confidence Intervals for Measures of Interaction”, Epidemiology, 7(3), 286–290.

    Article  Google Scholar 

  3. BACHE, K., and LICHMAN, M. (2013), “UCI Machine Learning Repository”, University of California, Irvine, School of Information and Computer Sciences, http://archive.ics.uci.edu/ml/.

  4. BALLI, H.O., and SORENSON, B.E. ( 2013), “Interaction Effects in Econometrics”, Empirical Economics, 45, 583–603.

    Article  Google Scholar 

  5. BERRINGTON DE GONZÁLEZ, A., and COX, D.R. (2007), “Interpretation of Interaction: A Review”, The Annals of Applied Statistics, 1(2), 371–385.

    MathSciNet  Article  MATH  Google Scholar 

  6. BREIMAN, L. (1996), “Bagging Predictors”, Machine Learning, 24(2), 123–140.

    MATH  Google Scholar 

  7. BREIMAN, L. (2001), “Random Forests”, Machine Learning, 45(1), 5–32.

    Article  MATH  Google Scholar 

  8. BREIMAN, L., FRIEDMAN, J., OLSHEN, R., and STONE, C. (1984), Classification and Regression Trees, Monterey CA: Wadsworth and Brooks.

    Google Scholar 

  9. CHIPMAN, H.A., GEORGE, E.I., and MCCULLOCH, R.E. (2010), “Bart: Bayesian Additive Regression Trees”, The Annals of Applied Statistics, 4(1), 266–298.

    MathSciNet  Article  MATH  Google Scholar 

  10. COHEN, J., COHEN, P., WEST, S., and AIKEN, L. (2003), Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, Mahwah NJ: Lawrence Erlbaum.

    Google Scholar 

  11. CULP, M., JOHNSON, K., and MICHAILIDIS, G. (2012), ada: An R Package for Stochastic Boosting, R package version 2.0-3, http://CRAN.R-project.org/package=ada.

  12. DAWSON, J. (2014), “Moderation in Management Research: What, Why, When, and How”, Journal of Business and Psychology, 29(1), 1–19.

    MathSciNet  Article  Google Scholar 

  13. DEMŠAR, J. (2006), “Statistical Comparisons of Classifiers Over Multiple Data Sets”, Journal of Machine Learning Research, 7, 1–30.

    MathSciNet  MATH  Google Scholar 

  14. DUSSELDORP, E., CONVERSANO, C., and VAN OS, B.J. (2010), “Combining an Additive and Tree-Based Regression Model Simultaneously: Stima”, Journal of Computational and Graphical Statistics, 19(3), 514–530.

    MathSciNet  Article  Google Scholar 

  15. FLEMING, T., and HARRINGTON, D. (1991), Counting Processes and Survival Analysis, Hoboken NJ: John Wiley and Sons, Inc.

    Google Scholar 

  16. FRIEDMAN, J.H. (1991), “Multivariate Adaptive Regression Splines”, The Annals of Statistics, 19(1), 1–67.

    MathSciNet  Article  MATH  Google Scholar 

  17. FRIEDMAN, J.H., HASTIE, T., and TIBSHIRANI, R. (2000), “Additive Logistic Regression: A Statistical View of Boosting”, The Annals of Statistics, 28(2), 337–374.

    MathSciNet  Article  MATH  Google Scholar 

  18. GRUBINGER, T., ZEILEIS, A., and PFEIFFER, K.P. (2011), “Evolutionary Learning of Globally Optimal Classification and Regression Trees in R”,Working Paper 2011-20. Working Papers in Economics and Statistics, Research Platform Empirical and Experimental Economics, Universitaet Innsbruck, http://EconPapers.RePEc.org/RePEc:inn:wpaper:2011-20.

  19. HALVORSEN, T. (2012), “ElemStatLearn: Data Sets, Functions and Examples”, in The Elements of Statistical Learning, Data Mining, Inference, and Prediction, T. Hastie, R. Tibshirani, and J. Friedman, R package version 2012.04-0, http://CRAN.R-project.org/package=ElemStatLearn.

  20. HAND, D.J. (1997), Construction and Assessment of Classification Rules, Chichester: Wiley.

    Google Scholar 

  21. HASTIE, T.J., FRIEDMAN, J.H., and TIBSHIRANI, R.J. (2009), Elements of Statistical Learning, New York: Springer.

    Google Scholar 

  22. HASTIE, T.J., and TIBSHIRANI, R.J. (1990), Generalized AdditiveModels, London: Chapman and Hall.

    Google Scholar 

  23. HASTIE, T. (2013), gam: Generalized Additive Models, R package version 1.09, http://CRAN.R-project.org/package=gam.

  24. HASTIE, T., and TIBSHIRANI, R. (2013), mda: Mixture and Flexible Discriminant Analysis, R package version 0.4-4,S original by Hastie and Tishirani, original R port by Leisch, Hornik, and Ripley, http://CRAN.R-project.org/package=mda.

  25. HOTHORN, T., HORNIK, K., and ZEILEIS, A. (2006) , “Unbiased Recursive Partitioning: A Conditional Inference Framework”, Journal of Computational and Graphical Statistics, 15(3), 651–674.

    MathSciNet  Article  Google Scholar 

  26. HOSMER, D.W., LEMESHOW, S., and STURDIVANT, R.X. (2013), Applied Logistic Regression (3rd. ed.), Hoboken, NJ: John Wiley and Sons, Inc.

  27. HOSMER, D.W., and LEMESHOW, S. (2000), Applied Logistic Regression (2nd ed.), Hoboken NJ: John Wiley and Sons, Inc.

  28. HOSMER, D.W., and LEMESHOW, S. (1989), Applied Logistic Regression (1st ed.), Hoboken NJ: John Wiley and Sons, Inc.

  29. HOTHORN, T. (2014), “TH.Data: TH’s Data Archive”, R package version 1.0-3, http://CRAN.R-project.org/package=TH.data.

  30. KALBFLEISCH, J.D., and PRENTICE, R.L.(1980), The Statistical Analysis of Failure Time Data, New York: John Wiley and Sons.

    Google Scholar 

  31. KAPELNER, A., and BLEICH, J. (2013), “bartMachine: A Powerful Tool for Machine Learning”, ArXiv e-prints, http://arxiv.org/abs/1312.2171.

  32. KARATZOGLOU, A., MEYER, D., and HORNIK, K. (2006), “Support Vector Machines in R”, Journal of Statistical Software, 15(9), 1–28.

    Article  Google Scholar 

  33. KARATZOGLOU, A., SMOLA, A., HORNIK, K., and ZEILEIS, A. (2004), “kernlab - An S4 Package for Kernel Methods in R”, Journal of Statistical Software, 11(9), 1–20.

    Article  Google Scholar 

  34. KIM, H., and LOH, W. (2001), ‘Classification TreesWith Unbiased Multiway Splits’, Journal of the American Statistical Association, 96, 589–604.

    MathSciNet  Article  Google Scholar 

  35. KUHN, A. [Contributions from J. Wing, S. Weston, A. Williams, C. Keefer, A. Engelhardt, T. Cooper, Z. Mayer and the R Core Team] (2014), “caret: Classification and Regression Training. R package version 6.0-30”, http://CRAN.R-project.org/package=caret.

  36. KUHN, M., WESTON, S., and COULTER, N. (2014), “C50: C5.0 Decision Trees and Rule-Based Models”, R package version 0.1.0-19, (C code for C5.0 by R. Quinlan), http://CRAN.R-project.org/package=C50.

  37. LANDWEHR, N., HALL, M. and FRANK, E. (2005), “Logistic Model Trees”, Machine Learning, 59(1-2), 161–205.

    Article  MATH  Google Scholar 

  38. LEISCH, F., and DIMITRIADU, E. (2010), “mlbench: Machine Learning Benchmark Problems”, R package version 2.1-1, http://CRAN.R-project.org/package=mlbench.

  39. LIAW, A., and WIENER, M. (2002), “Classification and Regression by randomForest”, R News, 2(3), 18–22.

    Google Scholar 

  40. LOH, W.-Y. (2009), “Improving the Precision of Classification Trees”, The Annals of Applied Statistics, 3(4), 1710–1737.

    MathSciNet  Article  MATH  Google Scholar 

  41. LOH, W.-Y., and SHIH, Y.-S. (1997), “Split Selection Methods for Classification Trees”, Statistica Sinica, 7(4), 815–840.

    MathSciNet  MATH  Google Scholar 

  42. MCCULLAGH, P., and NELDER, J.A. (1989), Generalized Linear Models, London: Chapman and Hall.

    Google Scholar 

  43. MCFADDEN, D. (1974), “The Measurement of Urban Travel Demand”, Journal of Public Economics, 3(4), 303–328.

    Article  Google Scholar 

  44. MENARD, S. (2000), “Coefficients of Determination for Multiple Logistic Regression Analysis”, The American Statistician, 54(1), 17–24.

    Google Scholar 

  45. MORGAN, J.N., and SONQUIST, J.A. (1963), “Problems in the Analysis of Survey Data, and a Proposal”, Journal of the American Statistical Association, 58(302), 415–434.

    Article  MATH  Google Scholar 

  46. NAGLER, J. (1993), “Scobit: An Alternative Estimator to Logit and Probit”, American Journal of Political Science, 38(1), 230–255.

    Article  Google Scholar 

  47. NELDER, J.A., and WEDDERBURN, R.W.M. (1972), “Generalized Linear Models”, Journal of the Royal Statistical Society, Series A, 135, 370–384.

    Article  Google Scholar 

  48. PETERS, A., and HOTHORN, T. (2013), “ipred: Improved Predictors”, R package version 0.9-3, http://CRAN.R-project.org/package=ipred.

  49. QUINLAN, J.R. (1993), C4.5: Programs for Machine Learning, San Franciso CA: Morgan Kaufmann Publishers Inc.

  50. R CORE TEAM (2016), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria.

    Google Scholar 

  51. RIPLEY, B.D. (1996), Pattern Recognition and Neural Networks, New York NY: Cambridge University Press.

  52. RUSCH, T., LEE, I., HORNIK, K., JANK, W., and ZEILEIS, A. (2013), “Influencing Elections with Statistics: Targeting Voters with Logistic Regression Trees”, The Annals of Applied Statistics, 7(3), 1612–1639.

    MathSciNet  Article  MATH  Google Scholar 

  53. THERNEAU, T., ATKINSON, B., and RIPLEY, B.D. (2014). rpart: Recursive Partitioning and Regression Trees. R package version 4.1-5, http://CRAN.R-project.org/package=rpart.

  54. VENABLES, W.N., and RIPLEY, B.D. (2002) Modern Applied Statistics with S, NewYork: Springer.

    Google Scholar 

  55. WITTEN, I.H., and FRANK, E. (2005), Data Mining: Practical Machine Learning Tools and Techniques, San Francisco CA: Morgan Kaufmann.

    Google Scholar 

  56. ZEILEIS, A., HOTHORN, T., and HORNIK, K.(2008), “Model-Based Recursive Partitioning”, Journal of Computational and Graphical Statistics, 17(2), 492–514.

    MathSciNet  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Claudio Conversano.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Conversano, C., Dusseldorp, E. Modeling Threshold Interaction Effects Through the Logistic Classification Trunk. J Classif 34, 399–426 (2017). https://doi.org/10.1007/s00357-017-9241-y

Download citation

Keywords

  • STIMA
  • Generalized linear modeling
  • Logistic Regression
  • Recursive partitioning
  • Interaction effects
  • Regression trunk