Machine Learning

, Volume 46, Issue 1–3, pp 291–314 | Cite as

A Simple Decomposition Method for Support Vector Machines

  • Chih-Wei Hsu
  • Chih-Jen Lin


The decomposition method is currently one of the major methods for solving support vector machines. An important issue of this method is the selection of working sets. In this paper through the design of decomposition methods for bound-constrained SVM formulations we demonstrate that the working set selection is not a trivial task. Then from the experimental analysis we propose a simple selection of the working set which leads to faster convergences for difficult cases. Numerical experiments on different types of problems are conducted to demonstrate the viability of the proposed method.

support vector machines decomposition methods classification 


  1. Blake, C. L. & Merz, C. J. (1998). UCI repository of machine learning databases. Irvine, CA. (Available at Scholar
  2. Chang, C.-C., Hsu, C.-W., & Lin, C.-J. (2000). The analysis of decomposition methods for support vector machines. IEEE Trans. Neural Networks, 11:4, 1003-1008.Google Scholar
  3. Cristianini, N. & Shawe-Taylor, J. (2000). An introduction to support vector machines. Cambridge, UK: Cambridge University Press.Google Scholar
  4. Fletcher, R. (1987). Practical methods of optimization. New York: John Wiley and Sons.Google Scholar
  5. Friess, T.-T., Cristianini, N., & Campbell, C. (1998). The kernel adatron algorithm: A fast and simple learning procedure for support vector machines. In Proceeding of 15th Intl. Conf. Machine Learning. San Francisco, CA: Morgan Kaufman Publishers.Google Scholar
  6. Ho, T. K. & Kleinberg, E. M. (1996). Building projectable classifiers of arbitrary complexity. In Proceedings of the 13th International Conference on Pattern Recognition (pp. 880-885). Vienna, Austria.Google Scholar
  7. Joachims, T. (1998). Making large-scale SVM learning practical. In B. Schölkopf, C. J. C. Burges, & A. J. Smola (Eds.). Advances in kernel methods-support vector learning. Cambridge, MA: MIT Press.Google Scholar
  8. Joachims, T. (2000). Private communication.Google Scholar
  9. Laskov, P. (2002). Feasible direction decomposition algorithms for training support vector machines, Machine Learning, 46, 315-349.Google Scholar
  10. Lin, C.-J. (2000). On the convergence of the decomposition method for support vector machines (Technical Report). Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan. To appear in IEEE Trans. Neural Network.Google Scholar
  11. Lin, C.-J. & Moré, J. J. (1999). Newton's method for large-scale bound constrained problems. SIAM J. Optim., 9, 1100-1127. (Software available at Scholar
  12. Mangasarian, O. L. & Musicant, D. R. (1999). Successive overrelaxation for support vector machines. IEEE Trans. Neural Networks, 10:5, 1032-1037.Google Scholar
  13. Michie, D., Spiegelhalter, D. J., & Taylor, C. C. (1994). Machine learning, neural and statistical classification.Google Scholar
  14. Englewood Cliffs, N.J.: Prentice Hall. (Data available at anonymous ftp: Scholar
  15. Osuna, E., Freund, R., & Girosi, F. (1997). Training support vector machines: An application to face detection. In Proceedings of CVPR'97.Google Scholar
  16. Platt, J. C. (1998). Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. J. C. Burges, & A. J. Smola (Eds.). Advances in kernel methods-support vector learning. Cambridge, MA: MIT Press.Google Scholar
  17. Powell, M. J. D. (1973). On search directions for minimization. Math. Programming, 4, 193-201.Google Scholar
  18. Saunders, C., Stitson, M. O., Weston, J., Bottou, L., Schölkopf, B., & Smola, A. (1998). Support vector machine reference manual (Technical Report No. CSD-TR-98-03). Egham, UK: Royal Holloway, University of London.Google Scholar
  19. Schölkopf, B., Burges, C. J. C., & Smola, A. J. (Eds.). (1998). Advances in kernel methods-support vector learning. Cambridge, MA: MIT Press.Google Scholar
  20. Vanderbei, R. (1994). LOQO: An interior point code for quadratic programming (Technical Report No. SOR 94-15). Statistics and Operations Research, Princeton University. (revised November, 1998)Google Scholar
  21. Vapnik, V. (1995). The nature of statistical learning theory. New York, NY: Springer-Verlag.Google Scholar
  22. Vapnik, V. (1998). Statistical learning theory. New York, NY: John Wiley.Google Scholar

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • Chih-Wei Hsu
    • 1
  • Chih-Jen Lin
    • 1
  1. 1.Department of Computer Science and Information EngineeringNational Taiwan UniversityTaipeiTaiwan, Republic of China

Personalised recommendations