Skip to main content
SpringerLink
Log in

Least squares structural twin bounded support vector machine on class scatter

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Several projects and application development teams are spending their precious time and energy in the field of classification and regression. So, the main target of the proposed model is to develop a computationally fast and efficient algorithm for binary classification problems, which also provides better generalization in various realistic applications. As we know that support vector machines (SVMs) obtain better generalization by considering the structural risk minimization (SRM) principle but suffers from computational cost and the twin version of SVM (TWSVM) finds faster learning time by following the empirical risk minimization (ERM) principle that compromises with the generalization. However, the impact of class distributions has not been discussed in either classical SVM or conventional TWSVM. To address this issue majorly, Peng and Xu, (2013) have proposed an approach named minimum class variance TWSVM (RMCV-TWSVM), which deals with the class information by considering a model of data uncertainty but missed out between class information. Here, we propose an efficient approach that improves the generalization performance and effectively handles the computational burden to follows the least-squares approach by incorporating the total within-class and between-class information for binary classification named least squares structural twin bounded support vector machine on class scatter (LS-STBSVM). All the computational results of approaches on several important benchmark UCI real-world datasets as well as KEEL artificial datasets by using a linear and non-linear kernel have been analyzed which shows that the proposed approach LS-STBSVM has a significant impact on both computational cost and generalization ability over other classification approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 3.1
Algorithm 3.2
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

We have mentioned the source of the data in the text.

References

  1. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    Article  MATH  Google Scholar 

  2. Harris T (2015) Credit scoring using the clustered support vector machine. Expert Syst Appl 42(2):741–750

    Article  Google Scholar 

  3. Mohammad AH, Alwada'n T, Al-Momani O (2016) Arabic text categorization using support vector machine, Naïve Bayes and neural network. GSTF J Comput (JoC) 5(1):108

    Article  Google Scholar 

  4. Zhang T, Chen W (2016) LMD based features for the automatic seizure detection of EEG signals using SVM. IEEE Trans Neural Syst Rehab Eng 25(8):1100–1108

    Article  Google Scholar 

  5. Hoang N-D, Nguyen Q-L, Bui DT (2018) Image processing–based classification of asphalt pavement cracks using support vector machine optimized by artificial bee colony. J Comput Civ Eng 32(5):04018037

    Article  Google Scholar 

  6. Liu Z-b, Zhou F-x, Qin Z-t, Luo X-g, Zhang J (2018) Classification of stellar spectra with SVM based on within-class scatter and between-class scatter. Astrophys Space Sci 363(7):140

    Article  MathSciNet  Google Scholar 

  7. Mohd N, Singh A, Bhadauria HS (2020) A novel SVM based IDS for distributed denial of sleep strike in wireless sensor networks. Wirel Pers Commun 111(3):1999–2022

    Article  Google Scholar 

  8. Soni G, Selvaradjou K (2020) Performance evaluation of wireless sensor network MAC protocols with early sleep problem. Int J Commun Netw Distrib Syst 25(2):123–144

    Google Scholar 

  9. Alirezaei M, Niaki STA, Niaki SAA (2019) A bi-objective hybrid optimization algorithm to reduce noise and data dimension in diabetes diagnosis using support vector machines. Expert Syst Appl 127:47–57

    Article  Google Scholar 

  10. Gupta D, Gupta U (2021a) On robust asymmetric Lagrangian ν-twin support vector regression using pinball loss function. Appl Soft Comput 102:107099

    Article  Google Scholar 

  11. Gupta U, Gupta D (2021b) On regularization based twin support vector regression with Huber loss. Neural Process Lett 53(1):459–515

    Article  Google Scholar 

  12. Suykens JAK, Vanderwalle J (1999) Least squares support vector machine classifier. Neural Process Lett 9:293–300

    Article  Google Scholar 

  13. Suykens J, Lukas L, Van Dooren P, Vandewalle J (1999) Least squares s support vector machine classifiers: a large-scale algorithm. In: Proceedings European Conference Circ.Th. Des. ECCTD'99 (Stresa Italy, du August 29 au September 2)

  14. Cauwenberghs G, Poggio T (2001) Incremental and decremental support vector machine learning. In: Advances in neural information processing systems, pp 409–415

  15. Lin C-F, Wang S-D (2002) Fuzzy support vector machines. IEEE Trans Neural Netwo 13(2):464–471

    Article  Google Scholar 

  16. Zhu J, Rosset S, Tibshirani R, Hastie TJ (2004) 1-norm support vector machines. In: Advances in neural information processing systems 16 (NIPS 2003), pp 49–56

  17. Zhang L, Zhou W, Jiao L (2004) Wavelet support vector machine. IEEE Trans Syst Man Cybern Part B (Cybern) 34(1):34–39

    Article  Google Scholar 

  18. Fung GM, Mangasarian OL (2005) Multicategory proximal support vector machine classifiers. Mach Learn 59(1–2):77–97

    Article  MATH  Google Scholar 

  19. Cervantes J, Li X, Wen Y, Li K (2008) Support vector machine classification for large data sets via minimum enclosing ball clustering. Neurocomputing 71(4–6):611–619

    Article  Google Scholar 

  20. Mangasarian OL, Wild EW (2006) Multisurface proximal support vector machine classification via generalized eigenvalues. IEEE Trans Pattern Anal Mach Intell 28(1):69–74

    Article  Google Scholar 

  21. Jayadeva R, Khemchandani R, Chandra S (2007) Twin support vector machines for pattern classification. IEEE Trans Pattern Anal Mach Intell 29(5):905–910

    Article  MATH  Google Scholar 

  22. Kumar MA, Gopal M (2009) Least squares s twin support vector machines for pattern classification. Expert Syst Appl 36(4):7535–7543

    Article  Google Scholar 

  23. Shao Y-H, Zhang C-H, Wang X-B, Deng N-Y (2011) Improvements on twin support vector machines. IEEE Trans Neural Netw 22(6):962–968

    Article  Google Scholar 

  24. Gupta U, Gupta D, Prasad M (2018) Kernel target alignment based fuzzy Least Square twin bounded support vector machine. In: 2018 IEEE symposium series on computational intelligence (SSCI). IEEE, pp 228–235

    Chapter  Google Scholar 

  25. Hazarika BB, Gupta D (2022a) Density weighted twin support vector machines for binary class imbalance learning. Neural Process Lett 54(2):1091–1130

    Article  Google Scholar 

  26. Gupta U, Gupta D (2019) Lagrangian twin-bounded support vector machine based on L2-norm. In: Kalita J, Balas V, Borah S, Pradhan R (eds) Recent Developments in Machine Learning and Data Analytics. Advances in Intelligent Systems and Computing, vol 740. Springer, Singapore. https://doi.org/10.1007/978-981-13-1280-9_40

  27. Zafeiriou S, Tefas A, Pitas I (2007) Minimum class variance support vector machines. IEEE Trans Image Process 16(10):2551–2564

    Article  MathSciNet  Google Scholar 

  28. Tefas A, Kotropoulos C, Pitas I (2001) Using support vector machines to enhance the performance of elastic graph matching for frontal face authentication. IEEE Trans Pattern Anal Mach Intell 23(7):735–746

    Article  Google Scholar 

  29. Fang B, Cheng M, Tang YY, He G (2009) Improving the discriminant ability of local margin based learning method by incorporating the global between-class separability criterion. Neurocomputing 73(1–3):536–541

    Article  Google Scholar 

  30. Chen X, Yang J, Ye Q, Liang J (2011) Recursive projection twin support vector machine via within-class variance minimization. Pattern Recogn 44(10–11):2643–2655

    Article  MATH  Google Scholar 

  31. Xue H, Chen S, Yang Q (2011) Structural regularized support vector machine: a framework for structural large margin classifier. IEEE Trans Neural Netw 22(4):573–587

    Article  Google Scholar 

  32. Yeung DS, Wang D, Ng WWY, Tsang ECC, Wang X (2007) Structured large margin machines: sensitive to data distributions. Mach Learn 68(2):171–200

    Article  MATH  Google Scholar 

  33. Hazarika BB, Gupta D (2022b) Improved twin bounded large margin distribution machines for binary classification. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-13738-7

  34. Jun G, Chung F-l, Wang S (2011) Matrix pattern-based minimum within-class scatter support vector machines. Appl Soft Comput 11(8):5602–5610

    Article  Google Scholar 

  35. Ye Q, Zhao C, Ye N (2012) Least squares twin support vector machine classification via maximum one-class within-class variance. Optim Methods Softw 27(1):53–69

    Article  MathSciNet  MATH  Google Scholar 

  36. Peng X, Dong X (2013) Robust minimum class variance twin support vector machine classifier. Neural Comput & Applic 22(5):999–1011

    Article  Google Scholar 

  37. Jiang Y, Frank HF, Leung. (2018) Fisher discriminant analysis with new between-class scatter matrix for audio signal classification. In: 2018 IEEE 23rd international conference on digital signal processing (DSP). IEEE, pp 1–5

    Google Scholar 

  38. Jimenez C, Alvarez AM, Orozco A (2018) A data representation approach to support imbalanced data classification based on TWSVM. In: Vera-Rodriguez R, Fierrez J, Morales A (eds) Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications. CIARP 2018. Lecture Notes in Computer Science, vol 11401. Springer, Cham. https://doi.org/10.1007/978-3-030-13469-3_7

  39. Liu L, Wang L, Ji H, Zang W, Li D (2017) Between-class discriminant twin support vector machine for imbalanced data classification. In: 2017 Chinese automation congress (CAC). IEEE, pp 7117–7122. https://doi.org/10.1109/CAC.2017.8244062

  40. Shao Y-H, Chen W-J, Zhang J-J, Wang Z, Deng N-Y (2014) An efficient weighted Lagrangian twin support vector machine for imbalanced data classification. Pattern Recogn 47(9):3158–3167

    Article  MATH  Google Scholar 

  41. Wang H, Yitian X, Zhou Z (2021) Twin-parametric margin support vector machine with truncated pinball loss. Neural Comput Appl 33:3781–3798. https://doi.org/10.1007/s00521-020-05225-7

  42. Hazarika BB, Gupta D (2021) Density-weighted support vector machines for binary class imbalance learning. Neural Comput & Applic 33(9):4243–4261

    Article  Google Scholar 

  43. Richhariya B, Tanveer M (2022) A fuzzy universum least squares twin support vector machine (FULSTSVM). Neural Comput Appl 34:11411–11422. https://doi.org/10.1007/s00521-021-05721-4

  44. Wan M, Wang Q, Liu H, Li X (2015) A new face recognition system based on kernel maximum between-class margin criterion (KMMC). In: 3rd international conference on mechatronics, robotics and automation. Atlantis Press

    Google Scholar 

  45. Richhariya B, Tanveer M (2018) A robust fuzzy least squares twin support vector machine for class imbalance learning. Appl Soft Comput 71:418–432

    Article  Google Scholar 

  46. UCI datasets repositories (2021) Available at: https://archive.ics.uci.edu/ml/datasets.phpVapnik, Vladimir. "Support-vector networks." Mach Learn 20 (1995): 273–297. Accessed 15 Oct 2020

  47. KEEL datasets repositories (2020). Available at: on https://sci2s.ugr.es/keel/datasets.php. Accessed 15 Oct 2020

  48. Triguero I, González S, Moyano JM, García S, Alcalá-Fdez J, Luengo J, Fernández A, del Jesús MJ, Sánchez L, Herrera F  (2017) KEEL 3.0: An Open Source Software for Multi-Stage Analysis in Data Mining. Int J Comput Intell Syst 10(1):1238. Published by Atlantis Press

  49. Alcalá-Fdez J, Fernández A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2011) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. J Multiple-Valued Logic Soft Comput 17:255–287

  50. Napierała K, Stefanowski J, Wilk S (2010) Learning from imbalanced data in presence of noisy and borderline examples. In: Szczuka M, Kryszkiewicz M, Ramanna S, Jensen R, Hu Q (eds) Rough Sets and Current Trends in Computing. RSCTC 2010. Lecture Notes in Computer Science, vol 6086. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13529-3_18

  51. Alcalá-Fdez J, Sanchez L, Garcia S, Jose M, del Jesus S, Ventura JM, Garrell JO et al (2009) KEEL: a software tool to assess evolutionary algorithms for data mining problems. Soft Comput 13(3):307–318

    Article  Google Scholar 

  52. Rastogi R, Sharma S, Chandra S (2018) Robust parametric twin support vector machine for pattern classification. Neural Process Lett 47(1):293–323

    Article  Google Scholar 

  53. Gupta U, Gupta D (2021c) Regularized based implicit Lagrangian twin extreme learning machine in primal for pattern classification. Int J Mach Learn Cybern:1–32

  54. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  55. Zheng M, Li T, Zhu R, Tang Y, Tang M, Lin L, Ma Z (2020) Conditional Wasserstein generative adversarial network-gradient penalty-based approach to alleviating imbalanced data classification. Inf Sci 512:1009–1023

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Bennett University, Greater Noida, India

    Umesh Gupta

  2. National Institute of Technology Arunachal Pradesh, Arunachal Pradesh, India

    Deepak Gupta

Authors
  1. Umesh Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deepak Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deepak Gupta.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

The total within-class scatter are Aϕ = ((ϕ(S) − e1S∗)T(ϕ(S) − e1S∗) + (ϕ(V) − e2V∗)T(ϕ(V) − e2V∗)) and Bϕ = ((S∗ − V∗)T(S∗ − V∗))are the total between class scatter; The mean vectors of “+” or “-” class data examples are \( {S}^{\ast }=\frac{1}{m_1}\sum \limits_{i=1}^{m_1}\phi \left({x}_i^T\right) \) and\( {V}^{\ast }=\frac{1}{m_2}\sum \limits_{i={m}_1+1}^{m_2}\phi \left({x}_i^T\right);{e}_1,{e}_2 \)are the column vectors of ones. The term \( \frac{c_3}{2}{w}_1^T\left({A}^{\phi }-{B}^{\phi}\right){w}_1 \)and\( \frac{c_4}{2}{w}_2^T\left({A}^{\phi }-{B}^{\phi}\right){w}_2 \) signifies the total within-class scatter and the total between class scatter in the objective function of LS-STBSVM. The derivation for within- class scatter is as follows:

$$ {\displaystyle \begin{array}{c}{w}_1^T{A}^{\phi }{w}_1={w}_1^T\left({\left(\phi (S)-{e}_1{S}^{\ast}\right)}^T\left(\phi (S)-{e}_1{S}^{\ast}\right)+{\left(\phi (V)-{e}_2{V}^{\ast}\right)}^T\left(\phi (V)-{e}_2{V}^{\ast}\right)\right){w}_1\\ {}={w}_1^T{\left(\phi (S)-{e}_1{S}^{\ast}\right)}^T\left(\phi (S)-{e}_1{S}^{\ast}\right){w}_1+{w}_1^T{\left(\phi (V)-{e}_2{V}^{\ast}\right)}^T\left(\phi (V)-{e}_2{V}^{\ast}\right){w}_1\end{array}} $$
(53)
$$ {\displaystyle \begin{array}{c}{w}_1^T{\left(\phi (S)-{e}_1{S}^{\ast}\right)}^T\left(\phi (S)-{e}_1{S}^{\ast}\right){w}_1={w}_1^T\left[\phi {(S)}^T\phi (S)-\phi {(S)}^T{e}_1{S}^{\ast }-{S}^{\ast^T}{e}_1^T\phi (S)+{e}_1^T{S}^{\ast^T}{e}_1{S}^{\ast}\right]{w}_1\\ {}={w}_1^T\left[\phi {(S)}^T\phi (S)-\frac{1}{m_1}\phi {(S)}^T{e}_1{e}_1^T\phi (S)-\frac{1}{m_1}\phi {(S)}^T{e}_1^T{e}_1\phi (S)+\frac{1}{m_1}\phi {(S)}^T{e}_1{e}_1^T\phi (S)\right]{w}_1\\ {}={w}_1^T\left[K{\left(S,{Q}^T\right)}^TK\left(S,{Q}^T\right)-\frac{1}{m_1}K{\left(S,{Q}^T\right)}^T{e}_1{e}_1^TK\left(S,{Q}^T\right)\right]{w}_1\end{array}} $$
(54)

Similarly

$$ {w}_1^T{\left(\phi (V)-{e}_1{V}^{\ast}\right)}^T\left(\phi (V)-{e}_1{V}^{\ast}\right){w}_1={w}_1^T\left[K{\left(V,{Q}^T\right)}^TK\left(V,{Q}^T\right)-\frac{1}{m_2}K{\left(V,{Q}^T\right)}^T{e}_2{e}_2^TK\left(V,{Q}^T\right)\right]{w}_1 $$
(55)

One can write Eq. (56) by using (53), (54) and (55)

$$ {A}^{\phi }=\left(\begin{array}{c}K\left({}^SK\left(S,{Q}^T\right)-\frac{1}{m_1}K\right({}^S{e}_1{e}_1^TK\left(S,{Q}^T\right)+\left(K\right({}^VK\left(V,{Q}^T\right)\\ {}-\frac{1}{m_2}K\Big({}^V{e}_2{e}_2^TK\left(V,{Q}^T\right)\end{array}\right) $$
(56)

The derivation for between class scatter is as follows:

$$ {\displaystyle \begin{array}{c}{w}_1^T{B}^{\phi }{w}_1={w}_1^T\left({\left({S}^{\ast }-{V}^{\ast}\right)}^T\left({S}^{\ast }-{V}^{\ast}\right)\right){w}_1\\ {}{w}_1^T{B}^{\phi }{w}_1={w}_1^T\left({S}^{\ast^T}{S}^{\ast }-{S}^{\ast^T}{V}^{\ast }-{V}^{\ast T}{S}^{\ast }+{V}^{\ast T}{V}^{\ast}\right){w}_1\\ {}\begin{array}{c}{w}_1^T{B}^{\phi }{w}_1={w}_1^T\left(\begin{array}{c}\frac{1}{m_1^2}\phi \left({}^S{e}_1{e}_1^T\phi (S)-\frac{1}{m_1\ast {m}_2}\phi \right({}^S{e}_1{e}_2^T\phi (V)\\ {}-\frac{1}{m_1\ast {m}_2}\phi \left({}^V{e}_2{e}_1^T\phi (S)+\frac{1}{m_2^2}\phi \right({}^V{e}_2{e}_2^T\phi (V)\end{array}\right){w}_1\\ {}{w}_1^T{B}^{\phi }{w}_1={w}_1^T\left(\begin{array}{c}\frac{1}{m_1^2}K\left({}^S{e}_1{e}_1^TK\left(S,{Q}^T\right)-\frac{1}{m_1\ast {m}_2}K\right({}^S{e}_1{e}_2^TK\left(V,{Q}^T\right)\\ {}-\frac{1}{m_1\ast {m}_2}K\left({}^V{e}_2{e}_1^TK\left(S,{Q}^T\right)+\frac{1}{m_2^2}K\right({}^V{e}_2{e}_2^TK\left(V,{Q}^T\right)\end{array}\right){w}_1\\ {}{B}^{\phi }=\left(\begin{array}{c}\frac{1}{m_1^2}K\left({}^S{e}_1{e}_1^TK\left(S,{Q}^T\right)-\frac{1}{m_1\ast {m}_2}K\right({}^S{e}_1{e}_2^TK\left(V,{Q}^T\right)\\ {}-\frac{1}{m_1\ast {m}_2}K\left({}^V{e}_2{e}_1^TK\left(S,{Q}^T\right)+\frac{1}{m_2^2}K\right({}^V{e}_2{e}_2^TK\left(V,{Q}^T\right)\end{array}\right)\end{array}\end{array}} $$
(57)

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gupta, U., Gupta, D. Least squares structural twin bounded support vector machine on class scatter. Appl Intell 53, 15321–15351 (2023). https://doi.org/10.1007/s10489-022-04237-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04237-1

Keywords

Search

Navigation