Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Regularization feature selection projection twin support vector machine via exterior penalty


In the past years, non-parallel plane classifiers that seek projection direction instead of hyperplane for each class have attracted much attention, such as the multi-weight vector projection support vector machine (MVSVM) and the projection twin support vector machine (PTSVM). Instead of solving two generalized eigenvalue problems in MVSVM, PTSVM solves two related SVM-type problems to obtain the two projection directions by solving two smaller quadratic programming problems, similar to twin support vector machine. In order to suppress input space features, we propose a novel non-parallel classifier to automatically select significant features, called regularization feature selection projection twin support vector machine (RFSPTSVM). In contrast to the PTSVM, we first incorporate a regularization term to ensure the optimization problems are convex, and then replace all the terms with L1-norm ones. By minimizing an exterior penalty function of the linear programming problem and using a fast generalized Newton algorithm, our RFSPTSVM obtains very sparse solutions. For nonlinear case, the method utilizes minimal number of kernel functions. The experimental results on toy datasets, Myeloma dataset, several UCI benchmark datasets, and NDCC generated datasets show the feasibility and effectiveness of the proposed method.

This is a preview of subscription content, log in to check access.

Fig. 1


  1. 1.

    Burges C (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2:121–167

  2. 2.

    Cristianini N, Shawe-Taylor J (2000) An introduction to support vector machines: and other kernel-based learning methods. Cambridge University Press, Cambridge

  3. 3.

    Cortes C, Vapnik VN (1995) Support vector networks. Mach Learn 20:273–297

  4. 4.

    Demiriz A, Bennett KP, Breneman CM, Embrechts MJ (2001) Support vector machine regression in chemometrics. In: Computing science and statistics, proceedings of the 33rd symposium on the interface. American Statistical Association for the Interface Foundation of North America, Washington, DC

  5. 5.

    Osuna E, Freund R, Girosi F (1997) Training support vector machines: an application to face detection. In: Proceedings of the 1997 IEEE Computer Society conference on computer vision pattern recognition, pp 130–136

  6. 6.

    Jia G, Martinez AM (2009) Support vector machines in face recognition with occlusions. In: Proceedings of the 2009 IEEE conference on comput vision and pattern recognition, Miami, Florida, pp 136–141

  7. 7.

    Hotta Kazuhiro (2008) Robust face recognition under partial occlusion based on support vector machine with local Gaussian summation kernel. Image Vis Comput 26(11):1490–1498

  8. 8.

    Wang Zhenyu, Yang Wankou, Ben Xianye (2015) Low-resolution degradation face recognition over long distance based on CCA. Neural Comput Appl 26(7):1645–1652

  9. 9.

    Yang Wankou, Wang Zhenyu, Sun Changyin (2015) A collaborative representation based projections method for feature extraction. Pattern Recogn 48(1):20–27

  10. 10.

    Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. Machine Learning ECML-98:137–142

  11. 11.

    Cao L (2003) Support vector machines experts for time series forecasting. Neurocomputing 51:321–339

  12. 12.

    Chen X, Yang J, Liang J, Ye Q (2012) Recursive robust least squares support vector regression based on maximum correntropy criterion. Neurocomputing 97:63–73

  13. 13.

    Zhao Y, Zhao J, Zhao M (2013) Twin least squares support vector regression. Neurocomputing 118:225–236

  14. 14.

    Ben Xianye, Zhang Peng et al (2015) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput Appl. doi:10.1007/s00521-015-2031-8

  15. 15.

    Ben Xianye, Meng Weixiao et al (2013) Kernel coupled distance metric learning for gait recognition and face recognition. Neurocomputing 120:577–589

  16. 16.

    Du B, Zhang L (2015) Target detection based on a dynamic subspace. Pattern Recogn 47(1):344–358

  17. 17.

    Du B, Zhang L (2014) A discriminative metric learning based anomaly detection method. IEEE Trans Geosci Remote Sens 52(11):6844–6857

  18. 18.

    Fung G, Mangasarian OL (2001) Proximal support vector machine classifiers. In: Provost F, Srikant R (eds) Proceedings of the knowledge discovery and data mining, pp 77–86

  19. 19.

    Mangasarian O, Wild E (2006) Multisurface proximal support vector classification via generalize eigenvalues. IEEE Trans Pattern Anal Mach Intell 28(1):69–74

  20. 20.

    Jayadeva R, Khemchandani S (2007) Chandra, twin support vector machines for pattern classification. IEEE Trans Pattern Anal Mach Intell 29:905–910

  21. 21.

    Arun Kumar M, Gopal M (2009) Least squares twin support vector machines for pattern classification. Expert Syst Appl 36:7535–7543

  22. 22.

    Ye Q, Zhao C, Ye N, Chen Y (2010) Multi-weight vector projection support vector machines. Pattern Recogn Lett 31(13):2006–2011

  23. 23.

    Chen X, Yang J, Ye Q, Liang J (2011) Recursive projection twin support vector machine via within-class variance minimization. Pattern Recogn. doi:10.1016/j.patcog.2011.03.001

  24. 24.

    Shao Yuan-Hai, Wang Zhen, Chen Wei-Jie, Deng Nai-Yang (2013) A regularization for the projection twin support vector machine. Knowl Based Syst 37:203–210

  25. 25.

    Zhu J, Rosset S, Hastie T, Tibshirani R (2004) 1-norm support vector machines. In: Thrun S, Saul LK, Scholkopf BH (eds) Advances in neural information processing systems16–NIPS2003. MIT Press, Cambridge

  26. 26.

    Golub T, Slonim D, Tamayo P, Huard C, Gaasenbeek M, Mesirov J, Coller H, Loh M, Downing J, Caligiuri M (1999) Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286:531–536

  27. 27.

    Guyon I, Weston J, Barhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46:389–422

  28. 28.

    Zhou WD, Zhang L, Jiao LC (2002) Linear programming support vector machines. Pattern Recogn 35(12):2927–2936

  29. 29.

    Zou H (2007) An improved 1-norm SVM for simultaneous classification and variable selection. In: Proceedings of the eleventh international conference on artificial intelligence and statistics

  30. 30.

    Fung G, Mangasarian OL (2004) A feature selection Newton method for support vector machine classification. Comput Optim Appl 28(2):185–202

  31. 31.

    Mangasarian OL (2006) Exact 1-norm support vector machines via unconstrained convex differentiable minimization. J Mach Learn Res 7:1517–1530

  32. 32.

    Gao Shangbing, Ye Q, Ye N (2011) 1-norm least squares twin support vector machines. Neurocomputing 74:3590–3597

  33. 33.

    Bai L, Wang Z, Shao YH et al (2014) A novel feature selection method for twin support vector machine. Knowl Based Syst 59:1–8

  34. 34.

    Ye Q, Zhao C, Ye N, Zheng H, Chen X (2012) A feature selection method for nonparallel plane support vector machine classification. Optim Methods Softw 27(3):431–443

  35. 35.

    Guo J et al (2014) Feature selection for least squares projection twin support vector machine. NeuroComputing 144:174–183

  36. 36.

    Tao Y, Yang J (2010) Quotient vs. difference: comparison between the two discriminant criteria. Neurocomputing 73:1808–1817

  37. 37.

    Mangasarian OL (1994) Nonlinear programming. SIAM, Philadelphia

  38. 38.

    Mangasarian OL, Meyer RR (1979) Nonlinear perturbation of linear programs. SIAM J Control Optim 17(6):745–752

  39. 39.

    Blake C, Merz C (1998) UCI repository of machine learning databases, Department of Information and Computer Sciences, University of California, Irvine.

  40. 40.

    Page D, Zhan F, Cussens J, Waddell M, Hardin J, Barlogie B, Shaughnessy J Jr (2002) Comparative data mining for microarrays: a case study based on Multiple Myeloma. Technical Report 1453, Computer Sciences Department, University of Wisconsin

  41. 41.

    Thompson ME (2006) NDCC: normally distributed clustered datasets on cubues.

  42. 42.

    LeCun Y, Cortes C (2010) MNIST handwritten digit database.

Download references


The authors would like to thank the anonymous reviewers for their constructive comments and suggestions. This work was partially supported by the Natural Science Foundation of Jiangsu Province of China (Grant No. BK20140794), the China Postdoctoral Science Foundation (Grant No. 2014M551599), and the Fundamental Research Funds for the Central Universities (Grant No. 30916011326).

Author information

Correspondence to Jianhui Guo.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yi, P., Song, A., Guo, J. et al. Regularization feature selection projection twin support vector machine via exterior penalty. Neural Comput & Applic 28, 683–697 (2017).

Download citation


  • Projection twin support vector machine
  • Multi-weight vector projection support vector machine
  • Feature selection
  • Exterior penalty