Skip to main content
SpringerLink
Log in

Penalized Least Squares Classifier: Classification by Regression Via Iterative Cost-Sensitive Learning

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Least squares estimate that can directly obtain the analytical solution to minimize the mean square error (MSE) is one of the most effective regression tools. It has also been applied to many classification tasks due to its simplicity, clear physical and tractability. However, there is a fundamental contradiction between regression with continuous samples and classification with discrete category labels, i.e., the output metric spaces are different. Considering this contradiction in regression-based classification, this paper presents a new linear classifier, termed penalized least squares classifier (PLSC), which gradually adds the penalty on the distance between misclassified samples and decision boundary to the loss function. Fundamentally, the decision boundary is obtained by minimizing MSE with iterative cost-sensitive learning. Then, an enhanced nonlinear neural network classifier, PLSC-BP is formulated, in which the penalized least squares mechanism is applied to tune the learning strategy of the conventional neural network by adjusting the cost factor for each sample. Extensive experiments on six synthetic datasets and eleven publicly available datasets show that classification, which cannot be realized by traditional least squares, can be conducted using a regression algorithm with iterative cost-sensitive learning. The proposed enhanced algorithm outperforms the traditional neural network classifiers and other classifiers in terms of classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Pandey SK, Janghel RR (2021) Correction to: recent deep learning techniques, challenges and its applications for medical healthcare system: a review. Neural Process Lett 53:3829. https://doi.org/10.1007/s11063-021-10527-5

    Article  Google Scholar 

  2. Acharjya DP, Abraham A (2020) Rough computing - a review of abstraction, hybridization and extent of applications. Eng Appl Artif Intel 96:103924. https://doi.org/10.1016/j.engappai.2020.103924

    Article  Google Scholar 

  3. Zhang J-J, Zhong P (2020) Learning biased svm with weighted within-class scatter for imbalanced classification. Neural Process Lett 51:797–817. https://doi.org/10.1007/s11063-019-10096-8

    Article  Google Scholar 

  4. Rao T, Li X, Xu M (2020) Learning multi-level deep representations for image emotion classification. Neural Process Lett 51:2043–2061. https://doi.org/10.1007/s11063-019-10033-9

    Article  Google Scholar 

  5. Rao CR, Mitra SK (1971) Generalized inverse of a matrix and its applications. Oper Res Q. https://doi.org/10.1525/9780520325883-032

    Article  MATH  Google Scholar 

  6. Huang J, Ma S, Zhang C-H (2008) Adaptive lasso for sparse high-dimensional regression models. Stat Sin 18:1603–1618

    MathSciNet  MATH  Google Scholar 

  7. Ma X, Zhou B, Xue S (2021) A meshless hermite weighted least-square method for piezoelectric structures. Appl Math Comput 400:126073. https://doi.org/10.1016/j.amc.2021.126073

    Article  MathSciNet  MATH  Google Scholar 

  8. Acharjya DP, Kiruba BGG (2022) A rough set, formal concept analysis and sem-pls integrated approach towards sustainable wearable computing in the adoption of smartwatch. Sustain Comput Infor 33:100647. https://doi.org/10.1016/j.suscom.2021.100647

    Article  Google Scholar 

  9. Lian H, Liu J, Fan Z (2021) Distributed learning for sketched kernel regression. Neural Netw 143:368–376. https://doi.org/10.1016/j.neunet.2021.06.020

    Article  MATH  Google Scholar 

  10. Geng C, Chen S (2018) Metric learning-guided least squares classifier learning. IEEE Trans Neural Netw Learn Syst 29(12):6409–6414. https://doi.org/10.1109/TNNLS.2018.2830802

    Article  MathSciNet  Google Scholar 

  11. Guliyev NJ, Ismailov VE (2018) On the approximation by single hidden layer feedforward neural networks with fixed weights. Neural Netw 98:296–304. https://doi.org/10.1016/j.neunet.2017.12.007

    Article  MATH  Google Scholar 

  12. Kim M (2021) The generalized extreme learning machines: tuning hyperparameters and limiting approach for the moore-penrose generalized inverse. Neural Netw 144:591–602. https://doi.org/10.1016/j.neunet.2021.09.008

    Article  MATH  Google Scholar 

  13. Zhang P-B, Yang Z-X (2020) A new learning paradigm for random vector functional-link network: Rvfl+. Neural Netw 122:94–105. https://doi.org/10.1016/j.neunet.2019.09.039

    Article  Google Scholar 

  14. Chen CLP, Liu Z (2018) Broad learning system: An effective and efficient incremental learning system without the need for deep architecture. IEEE Trans Neural Netw Learn Syst 29(1):10–24. https://doi.org/10.1109/TNNLS.2017.2716952

    Article  MathSciNet  Google Scholar 

  15. Seifollahi S, Yearwood J, Ofoghi B (2012) Novel weighting in single hidden layer feedforward neural networks for data classification. Comput Math Appl 64(2):128–136. https://doi.org/10.1016/j.camwa.2012.01.042

    Article  MathSciNet  MATH  Google Scholar 

  16. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297. https://doi.org/10.1007/BF00994018

    Article  MATH  Google Scholar 

  17. Suykens J, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300. https://doi.org/10.1023/A:1018628609742

    Article  Google Scholar 

  18. Zadeh SG, Schmid M (2021) Bias in cross-entropy-based training of deep survival networks. IEEE Trans Pattern Anal Mach Intell 43(9):3126–3137. https://doi.org/10.1109/TPAMI.2020.2979450

    Article  Google Scholar 

  19. Yang Y, Huang S, Huang W, Chang X (2021) Privacy-preserving cost-sensitive learning. IEEE Trans Neural Netw Learn Syst 32(5):2105–2116. https://doi.org/10.1109/TNNLS.2020.2996972

    Article  MathSciNet  Google Scholar 

  20. Huang C, Li Y, Loy CC, Tang X (2020) Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans Pattern Anal Mach Intell 42(11):2781–2794. https://doi.org/10.1109/TPAMI.2019.2914680

    Article  Google Scholar 

  21. Fernando KRM, Tsokos CP (2021) Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks. IEEE Trans Neural Netws Learn Syst 99:1–12. https://doi.org/10.1109/TNNLS.2020.3047335

    Article  Google Scholar 

  22. Toh K-A, Eng H-L (2008) Between classification-error approximation and weighted least-squares learning. IEEE Trans Pattern Anal Mach Intell 30(4):658–669. https://doi.org/10.1109/TPAMI.2007.70730

    Article  Google Scholar 

  23. Lkski J (2010) Iteratively reweighted least squares classifier and its l2-and l1-regularized kernel versions. B Pol Acad Sci-Tech 58(1):171–182. https://doi.org/10.2478/v10175-010-0018-2

    Article  Google Scholar 

  24. Hastie TJ, Rosset S et al (2009) Multi-class adaboost. Stat Interface 2:349–360. https://doi.org/10.4310/SII.2009.v2.n3.a8

    Article  MathSciNet  MATH  Google Scholar 

  25. Lin TY, Goyal P et al (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):318–327. https://doi.org/10.1109/ICCV.2017.324

    Article  Google Scholar 

  26. Zhou Z-H, Liu X-Y (2006) Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans Knowl Data Eng 18(1):63–77. https://doi.org/10.1109/TKDE.2006.17

    Article  MathSciNet  Google Scholar 

  27. Fernando KRM, Tsokos CP (2021) Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks. IEEE Trans Neural Netws Learn Syst 99:1–12. https://doi.org/10.1109/TNNLS.2020.3047335

    Article  Google Scholar 

  28. Castro CL, Braga AP (2013) Novel cost-sensitive approach to improve the multilayer perceptron performance on imbalanced data. IEEE Trans Neural Netw Learn Syst 24(6):888–899. https://doi.org/10.1109/TNNLS.2013.2246188

    Article  Google Scholar 

  29. Fahlman SE, Touretzky DS, et al (1988) Faster-learning variations on back-propagation: an empirical study. In: Proceedings of connectionist models summer school, pp 38–51

  30. Schiffmann W, Joost M, Werner R (1994) Optimization of the back-propagation algorithm for training multilayer perceptrons. University of Koblenz, Landau

    Google Scholar 

  31. Levenberg K (1944) A method for the solution of certain non-linear problems in least squares. Quart Appl Math 54:164–168

    Article  MathSciNet  MATH  Google Scholar 

  32. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL (eds) Parallel distributed processing: explorations in the microstructure of cognition, vol 1. MIT Press, Cambridge, pp 318–362

    Chapter  Google Scholar 

  33. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7(85):2399–2434

    MathSciNet  MATH  Google Scholar 

  34. Fayed HA, Atiya AF (2019) Speed up grid-search for parameter selection of support vector machines. Appl Soft Comput 80:202–210. https://doi.org/10.1016/j.asoc.2019.03.037

    Article  Google Scholar 

  35. Liu L, Jiang H, He P, Chen W, Liu X, Gao J, Han J (2020) On the variance of the adaptive learning rate and beyond. arXiv:1908.03265

  36. Takase T, Oyama S, Kurihara M (2018) Effective neural network training with adaptive learning rate based on training loss. Neural Netw 101:68–78. https://doi.org/10.1016/j.neunet.2018.01.016

    Article  Google Scholar 

  37. Angiulli F, Basta S, Pizzuti C (2006) Distance-based detection and prediction of outliers. IEEE Trans Knowledge Data Eng 18(2):145–160. https://doi.org/10.1109/TKDE.2006.29

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Internet of Things Engineering, Jiangnan University, 1800 Lihu Avenue, Wuxi, 214122, Jiangsu, China

    Siyuan Zhang & Linbo Xie

Authors
  1. Siyuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linbo Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linbo Xie.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Xie, L. Penalized Least Squares Classifier: Classification by Regression Via Iterative Cost-Sensitive Learning. Neural Process Lett 55, 8809–8828 (2023). https://doi.org/10.1007/s11063-023-11178-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11178-4

Keywords

Search

Navigation