Nyström Approximate Model Selection for LSSVM

  • Lizhong Ding
  • Shizhong Liao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7301)


Model selection is critical to least squares support vector machine (LSSVM). A major problem of existing model selection approaches is that a standard LSSVM needs to be solved with O(n 3) complexity for each iteration, where n is the number of training examples. In this paper, we propose an approximate approach to model selection of LSSVM. We use Nyström method to approximate a given kernel matrix by a low rank representation of it. With such approximation, we first design an efficient LSSVM algorithm and theoretically analyze the effect of kernel matrix approximation on the decision function of LSSVM. Based on the matrix approximation error bound of Nyström method, we derive a model approximation error bound, which is a theoretical guarantee of approximate model selection. We finally present an approximate model selection scheme, whose complexity is lower than the previous approaches. Experimental results on benchmark datasets demonstrate the effectiveness of approximate model selection.


model selection Nyström method matrix approximation least squares support vector machine 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cawley, G.C., Talbot, N.L.C.: Fast exact leave-one-out cross-validation of sparse least-squares support vector machines. Neural Networks 17(10), 1467–1475 (2004)zbMATHCrossRefGoogle Scholar
  2. 2.
    Cawley, G.C., Talbot, N.L.C.: Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters. Journal of Machine Learning Research 8, 841–861 (2007)zbMATHGoogle Scholar
  3. 3.
    Cawley, G.C., Talbot, N.L.C.: On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research 11, 2079–2107 (2010)MathSciNetGoogle Scholar
  4. 4.
    Chapelle, O., Vapnik, V.: Model selection for support vector machines. In: Advances in Neural Information Processing Systems, vol. 12, pp. 230–236. MIT Press, Cambridge (2000)Google Scholar
  5. 5.
    Chapelle, O., Vapnik, V., Bousquet, O., Mukherjee, S.: Choosing multiple parameters for support vector machines. Machine Learning 46(1), 131–159 (2002)zbMATHCrossRefGoogle Scholar
  6. 6.
    Cortes, C., Mohri, M., Talwalkar, A.: On the impact of kernel approximation on learning accuracy. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), Sardinia, Italy, pp. 113–120 (2010)Google Scholar
  7. 7.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)zbMATHGoogle Scholar
  8. 8.
    Drineas, P., Mahoney, M.: On the Nyström method for approximating a Gram matrix for improved kernel-based learning. Journal of Machine Learning Research 6, 2153–2175 (2005)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Duan, K., Keerthi, S., Poo, A.: Evaluation of simple performance measures for tuning SVM hyperparameters. Neurocomputing 51, 41–59 (2003)CrossRefGoogle Scholar
  10. 10.
    Golub, G., Van Loan, C.: Matrix Computations. Johns Hopkins University Press, Baltimore (1996)zbMATHGoogle Scholar
  11. 11.
    Guyon, I., Saffari, A., Dror, G., Cawley, G.: Model selection: Beyond the Bayesian / frequentist divide. Journal of Machine Learning Research 11, 61–87 (2010)MathSciNetGoogle Scholar
  12. 12.
    Higham, N.: Accuracy and stability of numerical algorithms. SIAM, Philadelphia (2002)zbMATHCrossRefGoogle Scholar
  13. 13.
    Kumar, S., Mohri, M., Talwalkar, A.: Sampling techniques for the Nyström method. In: Proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS), Clearwater, Florida, USA, pp. 304–311 (2009)Google Scholar
  14. 14.
    Luntz, A., Brailovsky, V.: On estimation of characters obtained in statistical procedure of recognition. Technicheskaya Kibernetica 3 (1969) (in Russian)Google Scholar
  15. 15.
    Rätsch, G., Onoda, T., Müller, K.: Soft margins for AdaBoost. Machine Learning 42(3), 287–320 (2001)zbMATHCrossRefGoogle Scholar
  16. 16.
    Suykens, J., Vandewalle, J.: Least squares support vector machine classifiers. Neural Processing Letters 9(3), 293–300 (1999)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Vapnik, V., Chapelle, O.: Bounds on error expectation for support vector machines. Neural Computation 12(9), 2013–2036 (2000)CrossRefGoogle Scholar
  18. 18.
    Vapnik, V.: Statistical Learning Theory. John Wiley & Sons, New York (1998)zbMATHGoogle Scholar
  19. 19.
    Williams, C., Seeger, M.: Using the Nyström method to speed up kernel machines. In: Advances in Neural Information Processing Systems 13, pp. 682–688. MIT Press, Cambridge (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Lizhong Ding
    • 1
  • Shizhong Liao
    • 1
  1. 1.School of Computer Science and TechnologyTianjin UniversityTianjinChina

Personalised recommendations