Abstract
The article presents a general view of a class of decomposition algorithms for training Support Vector Machines (SVM) which are motivated by the method of feasible directions. The first such algorithm for the pattern recognition SVM has been proposed in Joachims, T. (1999, Schölkopf et al. (Eds.) Advances in kernel methods-Support vector learning (pp. 185–208). MIT Press). Its extension to the regression SVM—the maximal inconsistency algorithm—has been recently presented by the author (Laskov, 2000, Solla, Leen, & Müller (Eds.) Advances in neural information processing systems 12 (pp. 484–490). MIT Press). A detailed account of both algorithms is carried out, complemented by theoretical investigation of the relationship between the two algorithms. It is proved that the two algorithms are equivalent for the pattern recognition SVM, and the feasible direction interpretation of the maximal inconsistency algorithm is given for the regression SVM. The experimental results demonstrate an order of magnitude decrease of training time in comparison with training without decomposition, and, most importantly, provide experimental evidence of the linear convergence rate of the feasible direction decomposition algorithms.
Article PDF
Similar content being viewed by others
References
Chang, C.-C., Hsu, C.-W., & Lin, C.-J. (1999). The analysis of decomposition methods for support vector machines. In Proceedings of the International Joint Conference on Artificial Intelligence. SVM Workshop.
Collobert, R. & Bengio, S. (2000). Support vector machines for large-scale regression problems. Technical Report IDIAP-RR 00-17, IDIAP.
Cristianini, N. & Shawe-Taylor, J. (2000). An introduction to support vector machines. Cambridge, UK: Cambridge University Press.
Hsu, C.-W. & Lin, C.-J. (1999). A simple decomposition method for support vector machines. Technical report, National Taiwan University.
Joachims, T. (1999). Making large-scale support vector machine learning practical. In Schölkopf et al. (Eds.). Advances in Kernel Methods-Support Vector Learning (pp. 169-184). Cambridge, MA: MIT Press.
Kaufmann, L. (1999). Solving the quadratic problem arising in support vector classification. In Schölkopf et al. (Eds.). Advances in Kernel Methods-Support Vector Learning (pp. 147-168). Cambridge, MA: MIT Press.
Keerthi, S. S., Shevade, S. K., Bhattacharyya, C., & Murthy, K. R. K. (1999). Improvements to Platt's SMO algorithm for SVM classifier design. Technical Report CD-99-14, National Institute of Singapure.
Laskov, P. (2000). An improved decomposition algorithm for regression support vector machines. In S. Solla, T. Leen, & K.-R. Müller (Eds.). Advances in neural information processing systems 12 (pp. 484-490). Cambridge, MA: MIT Press.
Mangasarian, O. L. (1969). Nonlinear programming. New York: McGraw-Hill.
Mangasarian, O. L. & Musicant, D. R. (1999a). Massive support vector regression. Technical Report 99-02, University of Wisconsin, Data Mining Institute.
Mangasarian, O. L. & Musicant, D. R. (1999b). Successive overrelaxation for support vector machines. Transactions on Neural Networks, 10:5, 100-106.
Nocedal, J. & Wright, S. J. (1999). Numerical optimization. New York, NY: Springer-Verlag.
Osuna, E. (1998). Support vector machines: Training and applications. Ph.D. Thesis, Massachusetts Institute of Technology.
Osuna, E., Freund, R., & Girosi, F. (1997). Improved training algorithm for support vector machines. Neural Networks and Signal Processing.
Platt, J. C. (1999). Fast training of support vector machines using sequential minimal optimization. In Schölkopf et al. (Eds.) 1999. Advances in kernel method-support vector training (pp. 185-208). Cambridge, MA: MIT Press.
Schölkopf B. (1997). Support vector learning. Ph.D. Thesis, Universität Tübingen.
Schölkopf B., Burges, C., & Smola, A. (Eds.) (1999). Advances in kernel methos-support vector learning. Cambridge, MA: MIT Press.
Shevade, S. K., Keerthi, S. S., Bhattacharyya, C., & Murthy, K. R. K. (1999). Improvements to SMO algorithm for SVM regression. Technical Report CD-99-16, National Institute of Singapure.
Smola, A. (1998). Learning with Kernels. Ph.D. Thesis, Technical University of Berlin.
Smola, A. & Schölkopf B. (1998). A tutorial on support vector regression. Technical Report NC2-TR-1998-030, NeuroCOLT2.
Vapnik, V. N. (1982). Estimation of dependences based on empirical data. New York, NY: Springer-Verlag.
Vapnik, V. N. (1995). The nature of statistical learning theory. New York, NY: Springer-Verlag.
Vapnik, V. N. (1999). Statistical learning theory. New York, NY: J. Wiley and Sons.
Zoutendijk, G. (1960). Methods of feasible directions. Amsterdam: Elsewier.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Laskov, P. Feasible Direction Decomposition Algorithms for Training Support Vector Machines. Machine Learning 46, 315–349 (2002). https://doi.org/10.1023/A:1012479116909
Issue Date:
DOI: https://doi.org/10.1023/A:1012479116909