Applied Intelligence

, Volume 23, Issue 3, pp 219–239 | Cite as

An Efficient Support Vector Machine Learning Method with Second-Order Cone Programming for Large-Scale Problems

  • Rameswar Debnath
  • Masakazu Muramatsu
  • Haruhisa Takahashi
Article

Abstract

In this paper we propose a new fast learning algorithm for the support vector machine (SVM). The proposed method is based on the technique of second-order cone programming. We reformulate the SVM's quadratic programming problem into the second-order cone programming problem. The proposed method needs to decompose the kernel matrix of SVM's optimization problem, and the decomposed matrix is used in the new optimization problem. Since the kernel matrix is positive semidefinite, the dimension of the decomposed matrix can be reduced by decomposition (factorization) methods. The performance of the proposed method depends on the dimension of the decomposed matrix. Experimental results show that the proposed method is much faster than the quadratic programming solver LOQO if the dimension of the decomposed matrix is small enough compared to that of the kernel matrix. The proposed method is also faster than the method proposed in (S. Fine and K. Scheinberg, 2001) for both low-rank and full-rank kernel matrices. The working set selection is an important issue in the SVM decomposition (chunking) method. We also modify Hsu and Lin's working set selection approach to deal with large working set. The proposed approach leads to faster convergence.

Keywords

second-order cone programming quadratic programming Cholesky factorization eigenvalue decomposition support vector machine 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. S. Fine and K. Scheinberg, “Efficient SVM training using low-rank kernel representations,” Journal of Machine Learning Research, vol. 2, pp. 243–264, 2001.Google Scholar
  2. C. Campbell and N. Cristianini, “Simple learning algorithms for training support vector machine,” Technical report, University of Bristol, 1998.Google Scholar
  3. V.N. Vapnik, Stasistical Learning Theory, Wiley: New York, 1998.Google Scholar
  4. R.J. Vanderbei, “Loqo: An interior point code for quadratic programming,” Tecnical report SOR 94-15, Princeton University, 1994.Google Scholar
  5. T. Joachims, “Making large-scale support vector machine learning practical,” in Advanvced in Kernel Methods: Support Vector Machine, edited by B. Schölkopf, C. Burges, and A. Smola, MIT Press: Cambridge, MA, 1998, pp. 169–184.Google Scholar
  6. E. Osuna, R. Freund, and F. Girosi, “An improved training algorithm for support vector machines,” in Proc. of IEEE'97, FL, 1997.Google Scholar
  7. J. Platt, “Fast training of support vector machines using sequential minimal optimization,” in Advanced in Kernel Methods: Support Vector Machine, edited by B. Schölkopf, C. Burges, and A. Smola, MIT Press: Cambridge, MA, 1998, pp. 185–208.Google Scholar
  8. C.-W. Hsu and C.-J. Lin, “A simple decomposition method for support vector machines,” Machine Learning, vol. 46, pp. 291–314, 2002.CrossRefMATHGoogle Scholar
  9. P. Laskov, “An improved decomposition algorithm for regression support vector machines,” Machine Learning, vol. 46, pp. 315–350, 2002.CrossRefMATHGoogle Scholar
  10. S.S. Kertee, S. Shevade, C. Bhattacharyya, and K. Murthy, “Improvements to Platt's SMO algorithm for SVM classifier design,” Neural Computation, vol. 13, no. 3, pp. 637–649, 2001.Google Scholar
  11. C.-C. Chang and C.-J. Lin, “Training ν-support vector cclassifiers: Theory and algorithm,” Neural Computation, vol. 13, no. 9, pp. 2119–2147, 2001.CrossRefMATHGoogle Scholar
  12. R. Collobert and S. Bengio, “SVMTorch: A support vector machine for large-scale regression and classification problems,” Journal of Machine Learning Research, vol. 1, pp. 143–160, 2001. Available at http://www.idiap.ch/learning/SVMTorch.html
  13. C.-C. Chang and C.-J. Lin, “LIMSVM: A library for support vector machines,” 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
  14. C.-J. Lin, “On the convergence of the decomposition method for support vector machines,” IEEE Trans. Neural Network, vol. 12, pp. 1288–1298, 2001.Google Scholar
  15. G.R.G. Lanckriet, N. Cristianini, P.L. Bartlett, L El Ghaoui, and M.I. Jordan, “Learning the kernel matrix with semidefinite programming,” Journal of Machine Learning Research, vol. 5, pp. 27–72, 2004.Google Scholar
  16. R.D.C. Monterio and T. Tsuchiya, “Polynomial convergence of primal-dual algorithms for the second-order cone programming based on the MZ-family of directions,” Math. Program., vol. 88, pp. 61–83, 2000.MathSciNetGoogle Scholar
  17. A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. MPS-SIAM Series on Optimization: Philadelphia, 2001.Google Scholar
  18. M. Muramatsu, “On a commutative class of search directions for linear programming over symmetric cones,” Journal of Optimization Theory and Applications, vol. 112, no. 3, pp. 595–625, 2002.CrossRefMATHMathSciNetGoogle Scholar
  19. R. Debnath, M. Muramatsu, and H. Takahashi, “The support vector machine learning using second order cone programming,” in Proc. IEEE Int. Joint Conference on Neural Networks, Budapest, Hungary, 25–29 July, 2004, pp. 2991–2996.Google Scholar
  20. R.D.C. Monteiro, “Primal-dual path following algorithms for semidefinite programming,” SIAM Journal on Optimization, vol. 7, pp. 663–678, 1997.CrossRefMATHMathSciNetGoogle Scholar
  21. C. Helmberg, F. Rendl, R.J. Vanderbei, and H. Wolkowicz, “An interior-point method for semidefinite programming,” SIAM Journal on Optimization, vol. 6, pp. 342–361, 1996.CrossRefMathSciNetMATHGoogle Scholar
  22. M. Kojima, S. Shindoh, and S. Hara, “Interior-point methods for the monotone linear complementary problem in symmetric matrices,” SIAM Journal on Optimization, vol. 7, pp. 86–125, 1997.CrossRefMathSciNetMATHGoogle Scholar
  23. E.D. Andersen, C. Roos, and T. Terlaky, “On implementing a primal-dual interior-point method for conic quadratic optimization,” Math. Programming Ser. B, vol. 95, pp. 249–277, 2003.MathSciNetCrossRefMATHGoogle Scholar
  24. Z. Cai, K.-C. Toh, “Solving second order cone programming via the augmented systems”. [online] Available: http://www.optimization-online.org/DB_HTML/2002/08/517.html
  25. S. Mehrotra, “On implementation of a primal-dual interior point method,” SIAM Journal on Optimization, vol. 2, no. 4, pp. 575–601, 1992.CrossRefMATHMathSciNetGoogle Scholar
  26. C.L. Blake and C.J. Merz, “UCI repository of machine learning databases,” Univ. California, Dept. Inform. Comp. Sc., Irvine, CA 1998. [online] Available: http://www.ics.uci.edu/~mlearn/MLRepository.html.
  27. G. Rätsch, Benchmark data sets. Available at http://www.first.gmd.de/~raetsch/data/benchmarks.htm.
  28. R. Debnath and H. Takahashi, “An improved working set selection method for SVM decomposition method,” in Proc. IEEE Int. Conference Intelligence Systems, Varna, Bulgaria, 21–24, 2004, pp. 520–523.Google Scholar
  29. C. Saunders, M.O. Stitson, J. Weston, L. Bottou, B. Schölkopf, and A. Smola, “Support vector machine reference manual,” Technical Report CSD-TR-98-03, Royal Holloway, University of London, Egham, UK, 1998.Google Scholar
  30. T. Joachims, Department of Computer Science, Cornell University, personal communication, 2003.Google Scholar
  31. W. Bress, W. Vetterling, S. Teukolsky, and B. Slannery, Numerical Receipes in C (The Art of Scientific Computing), 2nd ed. Cambridge University Press, 1992.Google Scholar
  32. G.H. Golub, C.F.V. Loan, Matrix Computations, 2nd ed. Johns Hopkins University Press, 1989.Google Scholar
  33. M.S. Bazaraa, C.M. Shetty, Nonlinear Programming: Theory and Algorithms, Wiley: New York, 1979.MATHGoogle Scholar
  34. J. Werner, Optimization-Theory and Applications, Vieweg, 1984.Google Scholar

Copyright information

© Springer Science + Business Media, Inc. 2005

Authors and Affiliations

  • Rameswar Debnath
    • 1
  • Masakazu Muramatsu
    • 2
  • Haruhisa Takahashi
    • 3
  1. 1.Department of Information and Communication EngineeringThe University of Electro-CommunicationsTokyoJapan
  2. 2.Department of Computer ScienceThe University of Electro-CommunicationsTokyoJapan
  3. 3.Department of Information and Communication EngineeringThe University of Electro-CommunicationsTokyoJapan

Personalised recommendations