A General Lp-norm Support Vector Machine via Mixed 0-1 Programming

  • Hai Thanh Nguyen
  • Katrin Franke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7376)

Abstract

Identifying a good feature subset that contributes most to the performance of Lp-norm Support Vector Machines (Lp-SVMs with p = 1 or p = 2) is an important task. We realize that the Lp-SVMs do not comprehensively consider irrelevant and redundant features, because the Lp-SVMs consider all n full-set features be important for training while skipping other 2 n  − 1 possible feature subsets at the same time. In previous work, we have studied the L1-norm SVM and applied it to the feature selection problem. In this paper, we extend our research to the L2-norm SVM and propose to generalize the Lp-SVMs into one general Lp-norm Support Vector Machine (GLp-SVM) that takes into account all 2 n possible feature subsets. We represent the GLp-SVM as a mixed 0-1 nonlinear programming problem (M01NLP). We prove that solving the new proposed M01NLP optimization problem results in a smaller error penalty and enlarges the margin between two support vector hyper-planes, thus possibly giving a better generalization capability of SVMs than solving the traditional Lp-SVMs. Moreover, by following the new formulation we can easily control the sparsity of the GLp-SVM by adding a linear constraint to the proposed M01NLP optimization problem. In order to reduce the computational complexity of directly solving the M01NLP problem, we propose to equivalently transform it into a mixed 0-1 linear programming (M01LP) problem if p = 1 or into a mixed 0-1 quadratic programming (M01QP) problem if p = 2. The M01LP and M01QP problems are then solved by using the branch and bound algorithm. Experimental results obtained over the UCI, LIBSVM, UNM and MIT Lincoln Lab datasets show that our new proposed GLp-SVM outperforms the traditional Lp-SVMs by improving the classification accuracy by more than 13.49%.

Keywords

support vector machine feature selection mixed 0-1 quadratic programming branch and bound 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bradley, P., Mangasarian, O.L.: Feature selection via concave minimization and support vector machines. In: Proceedings of the Fifteenth International Conference (ICML), pp. 82–90 (1998)Google Scholar
  2. 2.
    Mangasarian, O.L.: Exact 1-Norm Support Vector Machines Via Unconstrained Convex Differentiable Minimization (Special Topic on Machine Learning and Optimization). Journal of Machine Learning Research 7(2), 1517–1530 (2007)Google Scholar
  3. 3.
    Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., Vapnik, V.: Feature selection for SVMs. In: Advances in Neural Information Processing Systems, pp. 668–674 (2001)Google Scholar
  4. 4.
    Guyon, I., Weston, J., Barnhill, S., Vapnik, V.: Gene selection for cancer classification using support vector machines. Machine Learning 46(1), 389–422 (2002)MATHCrossRefGoogle Scholar
  5. 5.
    Guan, W., Gray, A., Leyffer, S.: Mixed-Integer Support Vector Machine. In: NIPS Workshop on Optimization for Machine Learning (2009)Google Scholar
  6. 6.
    Neumann, J., Schnorr, C., Steidl, G.: Combined SVM-based feature selection and classification. Machine Learning 61(1), 129–150 (2005)MATHCrossRefGoogle Scholar
  7. 7.
    Rakotomamonjy, A.: Variable selection using SVM based criteria. Journal of Machine Learning Research 3, 1357–1370 (2003)MathSciNetMATHGoogle Scholar
  8. 8.
    Chang, C.-T.: On the polynomial mixed 0-1 fractional programming problems. European Journal of Operational Research 131(1), 224–227 (2001)MathSciNetMATHCrossRefGoogle Scholar
  9. 9.
    Chang, C.-T.: An efficient linearization approach for mixed integer problems. European Journal of Operational Research 123, 652–659 (2000)MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer (1995)Google Scholar
  11. 11.
    Cortes, C., Vapnik, V.: Support-Vector Networks. In: Machine Learning, pp. 273-297 (1995)Google Scholar
  12. 12.
    Murphy, P.M., Aha, D.W.: UCI repository of machine learning databases. Technical report, Department of Information and Computer Science, University of California, Irvine (1992), http://www.ics.uci.edu/mlearn/MLRepository.html
  13. 13.
    TOMLAB, The optimization environment in MATLAB, http://tomopt.com/tomlab/
  14. 14.
    Guyon, I., Gunn, S., Nikravesh, M., Zadeh, L.A.: Feature Extraction: Foundations and Applications. STUDFUZZ. Physica-Verlag, Springer (2006)Google Scholar
  15. 15.
    Liu, H., Motoda, H.: Computational Methods of Feature Selection. Chapman & Hall/CRC (2008).Google Scholar
  16. 16.
    DMI Classification Software, http://www.cs.wisc.edu/dmi/
  17. 17.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), Data sets and software, http://www.csie.ntu.edu.tw/cjlin/libsvm/
  18. 18.
    Lippmann, R.P., Graf, I., Garfinkel, S.L., Gorton, A.S., Kendall, K.R., McClung, D.J., Weber, D.J., Webster, S.E., Wyschogrod, D., Zissman, M.A.: The 1998 DARPA/AFRL off-line intrusion detection evaluation. Presented to The First Intl. Work Workshop on Recent Advances in Intrusion Detection (RAID 1998) (No Printed Proceedings) Lovain-la-Neuve, Belgium, September 14-16 (1998)Google Scholar
  19. 19.
    UNM (University of New Mexico) audit data, http://www.cs.unm.edu/~immsec/systemcalls.htm
  20. 20.
    Bennett, K.P., Mangasarian, O.L.: Robust linear programming discrimination of two linearly inseparable sets. Optimization Methods and Software 1(1), 23–34 (1992)CrossRefGoogle Scholar
  21. 21.
    Zhu, J., Rosset, S., Hastie, T., Tibshirani, R.: 1-norm support vector machines. In: Neural Information Processing Systems (2003)Google Scholar
  22. 22.
    Wang, L., Xiatong, S.: On L1-Norm Multiclass Support Vector Machines: Methodology and Theory. Journal of the American Statistical Association 102, 583–594 (2007)MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Newman, R.C.: Computer Security: Protecting Digital Resources. Jones & Bartlett Learning (2009) ISBN 0763759945Google Scholar
  24. 24.
    Nguyen, H.T., Franke, K., Petrovi’c, S.: On General Definition of L1-norm Support Vector Machines for Feature Selection. The International Journal of Machine Learning and Computing 1(3), 279–283 (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Hai Thanh Nguyen
    • 1
  • Katrin Franke
    • 1
  1. 1.NISlab, Department of Computer Science and Media TechnologyGjøvik University CollegeGjøvikNorway

Personalised recommendations