Abstract
This paper addresses the problem of feature selection for Multi-class Support Vector Machines (MSVM). Basing on the l 0 and the l 2-l 0 regularization we consider two models for this problem. The l 0-norm is approximated by a suitable way such that the resulting optimization problems can be expressed as DC (Difference of Convex functions) programs for which DC programming and DC Algorithms (DCA) are investigated. The preliminary numerical experiments on real-world datasets show the efficiency and the superiority of our methods versus one of the best standard algorithms on booth feature selection and classification.
This research has been supported by ”Fonds Européens de Développement Régional” (FEDER) Lorraine via the project InnoMaD (Innovations techniques d’optimisation pour le traitement Massif de Données).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bradley, P.S., Mangasarian, O.L.: Eature selection via concave minimization and support vector machines. In: Shavlik, J. (ed.) Proceedings of the Fifteenth International Conferences Machine Learning (ICML1998), pp. 82–90. Morgan Kaufmann, San Francisco (1998)
Collobert, R., Sinz, F., Weston, J., Bottou, L.: Large scale transductive SVMs. J. Machine Learn. 7, 1687–1712 (2006)
Duan, K.B., Rajapakse, J.C., Wang, H., Azuaje, F.: Multiple SVM-RFE for Genne Selection in Cancer Classification With Expression Data. IEEE Transactions on NANOBIOSCIENCE 4, 228–234 (2005)
Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its Oracle Properties. Journal of the American Statistical Association 96, 1348–1360 (2001)
Guyon, I., Elisseeff, A.: An Introduction to Variable and Feature Selection. Journal of Machine Learning Research 3, 1157–1182 (2003)
Hui, Z.: The Adaptive Lasso and Its oracle Properties. Journal of the American Statistical Association 101(476), 1418–1429 (2006)
Huang, J., Ma, S., Zhang, C.H.: Adaptive Lasso for sparse high-dimentional regression models. Statistica Sinica 18, 1603–1618 (2008)
Le Thi, H.A., Belghiti, T., Pham, D.T.: A new efficient algorithm based on DC programming and DCA for Clustering. Journal of Global Optimization 37, 593–608 (2006)
Le Thi, H.A., Le Hoai, M., Pham, D.T.: Optimization based DC programming and DCA for Hierarchical Clustering. European Journal of Operational Research 183, 1067–1085 (2007)
Le Thi, H.A., Le Hoai, M., Nguyen, N.V., Pham, D.T.: A DC Programming approach for Feature Selection in Support Vector Machines learning. Journal of Advances in Data Analysis and Classification 2(3), 259–278 (2008)
Le Thi, H.A., Van Nguyen, V., Ouchani, S.: Gene selection for cancer classification using DCA. In: Tang, C., Ling, C.X., Zhou, X., Cercone, N.J., Li, X. (eds.) ADMA 2008. LNCS (LNAI), vol. 5139, pp. 62–72. Springer, Heidelberg (2008)
Le Thi, H.A., Pham, D.T.: The DC (Difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Annals of Operations Research 133, 23–46 (2005)
Le Thi, H.A.: DC Programming and DCA, http://lita.sciences.univ-metz.fr/~lethi/DCA.html
Liu, Y., Shen, X., Doss, H.: Multicategory ψ-Learning and Support Vector Machine: Computational Tools. Journal of Computational and Graphical Statistics 14, 219–236 (2005)
Liu, Y., Shen, X.: Multicategory ψ-Learning. Journal of the American Statistical Association 101, 500–509 (2006)
Neumann, J., Schnörr, C., Steidl, G.: SVM-based Feature Selection by Direct Objective Minimisation. In: Proc. of 26th DAGM Symposium Pattern Recognition, pp. 212–219 (2004)
Pham, D.T., Le Thi, H.A.: Convex analysis approach to d.c. programming: Theory, Algorithm and Applications. Acta Mathematica Vietnamica 22, 289–355 (1997)
Pham, D.T., Le Thi, H.: Optimization algorithms for solving the trust region subproblem. SIAMJ. Optimization 2, 476–505 (1998)
Ronan, C., Fabian, S., Jason, W., Lé, B.: Trading Convexity for Scalability. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, Pittsburgh, Pennsylvania, pp. 201–208 (2006)
Yuille, A.L., Rangarajan, A.: The Convex Concave Procedure. Neural Computation 15(4), 915–936 (2003)
Wang, L., Shen, X.: On l 1-norm multi-class support vector machine: methodology and theory. Journal of the American Statistical Association 102, 583–594 (2007)
Weston, J., Watkins, C.: Support Vector Machines for Multi-Class Pattern Recognition. In: Proceedings - European Symposium on Artificial Neural Networks, ESANN 1999, pp. 219–224. D-Facto public (1999)
Weston, J., Elisseeff, A., Schölkopf, B.: Use of Zero-Norm with Linear Models and Kernel Methods. Journal of Machine Learning Research 3, 1439–1461 (2003)
Zhang, H.H., Liu, Y., Wu, Y., Zhu, J.: Variable selection for the multicategory SVM via adaptive sup-norm regularization. Journal of Statistics 2, 149–167 (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer International Publishing Switzerland
About this paper
Cite this paper
Le Thi, H.A., Nguyen, M.C. (2013). Efficient Algorithms for Feature Selection in Multi-class Support Vector Machine. In: Nguyen, N., van Do, T., le Thi, H. (eds) Advanced Computational Methods for Knowledge Engineering. Studies in Computational Intelligence, vol 479. Springer, Heidelberg. https://doi.org/10.1007/978-3-319-00293-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-00293-4_4
Publisher Name: Springer, Heidelberg
Print ISBN: 978-3-319-00292-7
Online ISBN: 978-3-319-00293-4
eBook Packages: EngineeringEngineering (R0)