Abstract
The multinomial logit model is the most widely used model for the unordered multi-category responses. However, applications are typically restricted to the use of few predictors because in the high-dimensional case maximum likelihood estimates frequently do not exist. In this paper we are developing a boosting technique called multinomBoost that performs variable selection and fits the multinomial logit model also when predictors are high-dimensional. Since in multi-category models the effect of one predictor variable is represented by several parameters one has to distinguish between variable selection and parameter selection. A special feature of the approach is that, in contrast to existing approaches, it selects variables not parameters. The method can also distinguish between mandatory predictors and optional predictors. Moreover, it adapts to metric, binary, nominal and ordinal predictors. Regularization within the algorithm allows to include nominal and ordinal variables which have many categories. In the case of ordinal predictors the order information is used. The performance of boosting technique with respect to mean squared error, prediction error and the identification of relevant variables is investigated in a simulation study. The method is applied to the national Indonesia contraceptive prevalence survey and the identification of glass. Results are also compared with the Lasso approach which selects parameters.
Similar content being viewed by others
References
Blake C, Keogh E, Merz CJ (1998) UCI repository of machine learning databases
Bühlmann P (2006) Boosting for high-dimensional linear models. Ann Stat 34:559–583
Bühlmann P, Hothorn T (2007) Boosting algorithms: regularization, prediction and model fitting (with discussion). Stat Sci 22:477–505
Bühlmann P, Yu B (2003) Boosting with l2 loss: regression and classification. J Am Stat Assoc 98:324–339
Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. Machine Learning: Proceedings of the 13th international conference, pp 148–156
Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33:1–22
Friedman JH, Hastie T, Tibshirani R (2000) Additive logistic regression: a statistical view of boosting. Ann Stat 28:337–407
Gertheiss J, Tutz G (2009) Penalized regression with ordinal predictors. Int Stat Rev 77:345–365
Krishnapuram B, Carin L, Figueiredo MA, Hartemink AJ (2005) Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE Trans Pattern Anal Mach Intell 27:957–968
Lim TS, Loh WY, Shih YS (2000) A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach Learn 40:203–229
Meier L, van de Geer S, Bühlmann P (2008) The group lasso for logistic regression. J R Stat Soc 70:53–71
Nyquist H (1991) Restricted estimation of generalized linear models. J Appl Stat 40:133–141
Park MY, Hastie T (2007) L1-regularization path algorithm for generalized linear models. J R Stat Soc B 69:659–677
Schapire RE (1990) The strength of weak learnability. Mach Learn 5:197–227
Segerstedt B (1992) On ordinary ridge regression in generalized linear models. Commun Stat Theory Methods 21:2227–2246
Tibshirani R (1996) Regression shrinkage and selection via lasso. J R Stat Soc B 58:267–288
Tutz G, Binder H (2006) Generalized additive modelling with implicit variable selection by likelihood based boosting. Biometrics 62:961–971
Tutz G, Binder H (2007) Boosting ridge regression. Comput Stat Data Anal 51:6044–6059
Yuan M, Lin Y (2006) Model selection and estimation in regression with grouped variables. J R Stat Soc B 68:49–67
Zahid FM, Tutz G (2013) Ridge estimation for multinomial logit models with symmetric side constraints. Comput Stat 28(3): 1017–1034
Zhao P, Rocha G, Yu B (2009) The composite absolute penalties family for grouped and hierarchical variable selection. Ann Stat 37:3468–3497
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zahid, F.M., Tutz, G. Multinomial logit models with implicit variable selection. Adv Data Anal Classif 7, 393–416 (2013). https://doi.org/10.1007/s11634-013-0136-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11634-013-0136-4
Keywords
- False alarm rate
- Hit rate
- Likelihood-based boosting
- Logistic regression
- Multinomial logit
- Penalization
- Side constraints
- Variable selection