Neural Computing and Applications

, Volume 31, Supplement 2, pp 1045–1051 | Cite as

Generalization improvement for regularized least squares classification

  • Haitao GanEmail author
  • Qingshan She
  • Yuliang Ma
  • Wei Wu
  • Ming Meng
Original Article


In the past decades, regularized least squares classification (RLSC) is a commonly used supervised classification method in the machine learning filed because it can be easily resolved through the simple matrix analysis and achieve a close-form solution. Recently, some studies conjecture that the margin distribution is more crucial to the generalization performance. Moreover, from the view of margin distribution, RLSC only considers the first-order statistics (i.e., margin mean) and does not consider the actual higher-order statistics of margin distribution. In this paper, we propose a novel RLSC which takes into account the actual second-order (i.e., variance) information of margin distribution. It is intuitively expected that small margin variance will improve the generalization performance of RLSC from a geometric view. We incorporate the margin variance into the objective function of RLSC and achieve the optimal classifier by minimizing the margin variance. To evaluate the performance of our algorithm, we conduct a series of experiments on several benchmark datasets in comparison with RLSC, kernel minimum squared error, support vector machine and large margin distribution machine. And the empirical results verify the effectiveness of our algorithm and indicate that the margin distribution is helpful to improve the classification performance.


Regularized least squares classification Margin distribution Second-order statistic 



The work was supported by National Natural Science Foundation of China under Grant Nos. 61601162, 61501154 and 61671197, and Open Foundation of first level Zhejiang key in key discipline of Control Science and Engineering.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Adankon MM, Cheriet M (2011) Help-training for semi-supervised support vector machines. Pattern Recognit 44(9):2220–2230CrossRefGoogle Scholar
  2. 2.
    Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7:2399–2434MathSciNetzbMATHGoogle Scholar
  3. 3.
    Gao W, Zhou ZH (2013) On the doubt about margin explanation of boosting. Artif Intell 203:1–18MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Hsieh CJ, Chang KW, Lin CJ, Keerthi SS, Sundararajan S (2008) A dual coordinate descent method for large-scale linear SVM. In: Proceedings of the 25th international conference on machine learning, ACM, pp 408–415Google Scholar
  5. 5.
    Lichman M (2013) UCI machine learning repository.
  6. 6.
    Micchelli CA, Pontil M (2005) Learning the kernel function via regularization. J Mach Learn Res 6:1099–1125MathSciNetzbMATHGoogle Scholar
  7. 7.
    Reyzin L, Schapire RE (2006) How boosting the margin can also boost classifier complexity. In: Proceedings of the 23rd international conference on Machine learning, ACM, pp 753–760Google Scholar
  8. 8.
    Rifkin R, Yeo G, Poggio T (2003) Regularized least-squares classification. Nato Sci Ser Sub Ser III Comput Syst Sci 190:131–154Google Scholar
  9. 9.
    Vapnik VN, Vapnik V (1998) Statistical learning theory. Wiley, New YorkzbMATHGoogle Scholar
  10. 10.
    Wang L, Sugiyama M, Jing Z, Yang C, Zhou ZH, Feng J (2011) A refined margin analysis for boosting algorithms via equilibrium margin. J Mach Learn Res 12:1835–1863MathSciNetzbMATHGoogle Scholar
  11. 11.
    Wang Y, Chen S, Xue H, Fu Z (2015) Semi-supervised classification learning by discrimination-aware manifold regularization. Neurocomputing 147:299–306CrossRefGoogle Scholar
  12. 12.
    Xu J, Zhang X, Li Y (2001) Kernel MSE algorithm: a unified framework for KFD, LS-SVM and KRR. In: Proceedings of international joint conference on neural networks, IEEE, pp 1486–1491Google Scholar
  13. 13.
    Xue H, Chen S, Yang Q (2009) Discriminatively regularized least-squares classification. Pattern Recognit 42(1):93–104CrossRefzbMATHGoogle Scholar
  14. 14.
    Yang XJ, Wang L (2015) A modified Tikhonov regularization method. J Comput Appl Math 288:180–192MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Zhang P, Peng J (2004) SVM vs regularized least squares classification. In: Proceedings of the 17th international conference on pattern recognition, IEEE, pp 176–179Google Scholar
  16. 16.
    Zhou ZH (2014) Large margin distribution learning. In: Artificial neural networks in pattern recognition, Springer, pp 1–11Google Scholar

Copyright information

© The Natural Computing Applications Forum 2017

Authors and Affiliations

  1. 1.School of AutomationHangzhou Dianzi UniversityHangzhouChina

Personalised recommendations