Advertisement

A Fast Parallel Optimization for Training Support Vector Machine

  • Jian-xiong Dong
  • Adam Krzyżak
  • Ching_Y. Suen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2734)

Abstract

A fast SVM training algorithm for multi-classes consisting of parallel and sequential optimizations is presented. The main advantage of the parallel optimization step is to remove most non-support vectors quickly, which dramatically reduces the training time at the stage of sequential optimization. In addition, some strategies such as kernel caching, shrinking and calling BLAS functions are effectively integrated into the algorithm to speed up the training. Experiments on MNIST handwritten digit database have shown that, without sacrificing the generalization performance, the proposed algorithm has achieved a speed-up factor of 110, when compared with Keerthi et al.’s modified SMO. Moreover, for the first time ever we investigated the training performance of SVM on handwritten Chinese database ETL9B with more than 3000 categories and about 500,000 training samples. The total training time is just 5.1 hours. The raw error rate of 1.1% on ETL9B has been achieved.

Keywords

Support Vector Machine Kernel Matrix Radial Basic Function Kernel Sequential Minimization Optimization Training Support Vector Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Schölkopf, B., Burges, C.J.C., Vapnik, V.: Extracting support data for a given task. In: Proceedings of the First International Conference on Knowledge Discovery and Data Mining, Menlo Park, CA (1995) 252–257Google Scholar
  2. 2.
    DeCoste, D., Schölkopf, B.: Training invariant support vector machines. Machine Learning. 46(1–3) (2002) 161–190zbMATHCrossRefGoogle Scholar
  3. 3.
    Joachims, T.: Text categorization with support vector machine: learning with many relevant features. In: Proceedings of 10th European Conference on Machine Learning (ECML) (1998) 137–142Google Scholar
  4. 4.
    Osuna, E., Freund, R., Girosi, F.: Training support vector machines: An application to face detection. In: Proceedings of the 1997 conference on Computer Vision and Pattern Recognition(CVPR’97), Puerto Rico (1997) 130–136Google Scholar
  5. 5.
    Platt, J.C.: Fast training of support vector machines using sequential minimal optimization. In: Schölkopf, B., Burges, C.J.C., Smola, A. (eds.): Advances in kernel methods: Support Vector Machines, MIT Press, Cambridge, MA (1998) 185–208Google Scholar
  6. 6.
    Joachims T.: Making large-scale support vector machine learning practical. In: Schölkopf, B., Burges, C.J.C., Smola, A. (eds.): Advances in kernel methods: Support Vector Machines, MIT Press, Cambridge, MA (1998) 169–184Google Scholar
  7. 7.
    Dongarra, J.J., Croz, J.D., Duff, I.S., Hammarling, S.: A set of level 3 basic linear algebra subprograms. ACM Trans. Math. Soft. 16 (1990) 1–17zbMATHCrossRefGoogle Scholar
  8. 8.
    Kuhn, H., Tucker, A.: Nonlinear programming. In: Proceedings of 2nd Berkeley Symposium on Mathematical Statistics and Probabilistics. University of California Press (1951) 481–492Google Scholar
  9. 9.
    Dong, J.X., Suen, C.Y., Krzyżak, A.: A fast svm training algorithm. In: Lee, S.-W., Verri, A. (eds.): Pattern Recognition with Support Vector Machines. Springer Lecture Notes in Computer Science LNCS 2388, Niagara Falls, Canada (2002) 481–492CrossRefGoogle Scholar
  10. 10.
    Keerthi, S.S., Shevade, S.K., Bhattachayya, C., Murth K.R.K.: Improvements to Platt’s SMO algorithm for SVM classifier design. Neural Computation, 13 (2001) 637–649zbMATHCrossRefGoogle Scholar
  11. 11.
    Dong, J.X., Krzyżak, A., Suen, C.Y.: A fast svm training algorithm. International Journal of Pattern Recognition and Artificial Intelligence, 17(3) (2003) 1–18CrossRefGoogle Scholar
  12. 12.
    Saito, T., Yamada, H., Yamamoto, K.: An analysis of handprinted character database VIII: An estimation of the database ETL9 of handprinted characters in JIS Chinese characters by directional pattern matching approach. Bul. Electrotech 49(7) (1985) 487–525Google Scholar
  13. 13.
    Dong, J.X., Suen, C.Y., Krzyżak, A.: High accuracy handwritten Chinese character recognition using support vector machine. Tech. Rep. CENPARMI, Concordia University, Canada, (2003)Google Scholar
  14. 14.
    Yamada, H., Yamamoto, K., Saito, T.: A nonlinear normalization method for handprinted kanji character recognition-line density equalization. Pattern Recognition, 23(9) (1990) 1023–1029CrossRefGoogle Scholar
  15. 15.
    Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. John Wiley & Sons. Inc, New York (1973)zbMATHGoogle Scholar
  16. 16.
    Kimura, F., Wakabayashi T., Tsuruoka, S., Miyake, Y.: Improvement of handwritten Japanese character recognition using weighted direction code histogram. Pattern Recognition, 30(8) (1997) 1329–1337CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Jian-xiong Dong
    • 1
  • Adam Krzyżak
    • 2
  • Ching_Y. Suen
    • 1
  1. 1.Centre for Pattern Recognition and Machine IntelligenceConcordia UniversityMontrealCanada
  2. 2.Department of Computer ScienceConcordia UniversityMontrealCanada

Personalised recommendations