Sparse Least Squares Support Vector Machines by Forward Selection Based on Linear Discriminant Analysis

  • Shigeo Abe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5064)


In our previous work, we have developed sparse least squares support vector machines (sparse LS SVMs) trained in the reduced empirical feature space, spanned by the independent training data selected by the Cholesky factorization. In this paper, we propose selecting the independent training data by forward selection based on linear discriminant analysis in the empirical feature space. Namely, starting from the empty set, we add a training datum that maximally separates two classes in the empirical feature space. To calculate the separability in the empirical feature space we use linear discriminant analysis (LDA), which is equivalent to kernel discriminant analysis in the feature space. If the matrix associated with the LDA is singular, we consider that the datum does not contribute to the class separation and permanently delete it from the candidates of addition. We stop the addition of data when the objective function of LDA does not increase more than the prescribed value. By computer experiments for two-class and multi-class problems we show that in most cases we can reduce the number of support vectors more than with the previous method.


  1. 1.
    Suykens, J.A.K., Van Gestel, T., De Brabanter, J., De Moor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific Publishing, Singapore (2002)zbMATHGoogle Scholar
  2. 2.
    Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, Chichester (1998)zbMATHGoogle Scholar
  3. 3.
    Cawley, G.C., Talbot, N.L.C.: Improved sparse least-squares support vector machines. Neurocomputing 48, 1025–1031 (2002)CrossRefzbMATHGoogle Scholar
  4. 4.
    Valyon, J., Horváth, G.: A sparse least squares support vector machine classifier. In: Proc. IJCNN 2004, vol. 1, pp. 543–548 (2004)Google Scholar
  5. 5.
    Jiao, L., Bo, L., Wang, L.: Fast sparse approximation for least squares support vector machine. IEEE Trans. Neural Networks 18(3), 685–697 (2007)CrossRefGoogle Scholar
  6. 6.
    Smola, A.J., Bartlett, P.L.: Sparse greedy Gaussian process regression. Advances in Neural Information Processing Systems 13, 619–625 (2001)Google Scholar
  7. 7.
    Vincent, P., Bengio, Y.: Kernel matching pursuit. Machine Learning 48(1-3), 165–187 (2002)CrossRefzbMATHGoogle Scholar
  8. 8.
    Xiong, H., Swamy, M.N.S., Ahmad, M.O.: Optimizing the kernel in the empirical feature space. IEEE Trans. Neural Networks 16(2), 460–474 (2005)CrossRefGoogle Scholar
  9. 9.
    Abe, S.: Sparse least squares support vector training in the reduced empirical feature space. Pattern Analysis and Applications 10(3), 203–214 (2007)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Kaieda, K., Abe, S.: KPCA-based training of a kernel fuzzy classifier with ellipsoidal regions. International Journal of Approximate Reasoning 37(3), 145–253 (2004)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Abe, S.: Support Vector Machines for Pattern Classification. Springer, Heidelberg (2005)Google Scholar
  12. 12.
    Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Müller, K.-R.: Fisher discriminant analysis with kernels. In: Proc. NNSP 1999, pp. 41–48 (1999)Google Scholar
  13. 13.
    Ashihara, M., Abe, S.: Feature selection based on kernel discriminant analysis. In: Kollias, S., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4132, pp. 282–291. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Rätsch, G., Onoda, T., Müller, K.-R.: Soft margins for AdaBoost. Machine Learning 42(3), 287–320 (2001)CrossRefzbMATHGoogle Scholar
  15. 15.
  16. 16.
  17. 17.
    Abe, S.: Training of support vector machines with Mahalanobis kernels. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) ICANN 2005. LNCS, vol. 3697, pp. 571–576. Springer, Heidelberg (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Shigeo Abe
    • 1
  1. 1.Graduate School of EngineeringKobe UniversityKobeJapan

Personalised recommendations