Ensemble Approaches of Support Vector Machines for Multiclass Classification

  • Jun-Ki Min
  • Jin-Hyuk Hong
  • Sung-Bae Cho
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4815)


Support vector machine (SVM) which was originally designed for binary classification has achieved superior performance in various classification problems. In order to extend it to multiclass classification, one popular approach is to consider the problem as a collection of binary classification problems. Majority voting or winner-takes-all is then applied to combine those outputs, but it often causes problems to consider tie-breaks and tune the weights of individual classifiers. This paper presents two novel ensemble approaches: probabilistic ordering of one-vs-rest (OVR) SVMs with naïve Bayes classifier and multiple decision templates of OVR SVMs. Experiments with multiclass datasets have shown the usefulness of the ensemble methods.


Support vector machines Ensemble Naïve Bayes Multiple decision templates Cancer classification Fingerprint classification 


  1. 1.
    Cortes, C., Vapnik, V.: Support-Vector Networks. Machine Learning 20(3), 273–297 (1995)zbMATHGoogle Scholar
  2. 2.
    Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)Google Scholar
  3. 3.
    Bredensteiner, E., Bennett, K.: Multicategory Classification by Support Vector Machines. Computational Optimization and Applications. 12(1), 53–79 (1999)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Arenas-Garcia, J., Perez-Cruz, F.: Multi-Class Support Vector Machines: A New Approach. In: 2003 IEEE Int. Conf. Acoustics, Speech, and Signal Processing, pp. II-781–784 (2003) Google Scholar
  5. 5.
    Hsu, C., Lin, C.: A Comparison of Methods for Multiclass Support Vector Machines. IEEE Trans. Neural Networks. 13(2), 415–425 (2002)CrossRefGoogle Scholar
  6. 6.
    Rifkin, R.M., Klautau, A.: In Defense of One-Vs-All Classification. J. Machine Learning Research. 5, 101–141 (2004)MathSciNetGoogle Scholar
  7. 7.
    Lee, Y., Lin, Y., Wahba, G.: Multicategory Support Vector Machines. Tech. Rep. 1043, Dept. Statistics, Univ. of Wisconsin (2001) Google Scholar
  8. 8.
    Hong, J.-H., Cho, S.-B.: Multi-Class Cancer Classification with OVR-Support Vector Machines Selected by Naive Bayes Classifier. In: King, I., Wang, J., Chan, L., Wang, D. (eds.) ICONIP 2006. LNCS, vol. 4234, pp. 155–164. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  9. 9.
    Min, J.-K., Hong, J.-H., Cho, S.-B.: Effective Fingerprint Classification by Localized Models of Support Vector Machines. In: Zhang, D., Jain, A.K. (eds.) Advances in Biometrics. LNCS, vol. 3832, pp. 287–293. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Ramaswamy, S., et al.: Multiclass Cancer Diagnosis using Tumor Gene Expression Signatures. Proc. National Academy of Science. 98(26), 15149–15154 (2001)CrossRefGoogle Scholar
  11. 11.
    Jain, A.K., Prabhakar, S., Hong, L.: A Multichannel Approach to Fingerprint Classification. IEEE Trans. Pattern Analysis and Machine Intelligence 21(4), 348–359 (1999)CrossRefGoogle Scholar
  12. 12.
    Cho, S.-B., Ryu, J.: Classifying Gene Expression Data of Cancer using Classifier Ensemble with Mutually Exclusive Features. Proc. of the IEEE. 90(11), 1744–1753 (2002)CrossRefGoogle Scholar
  13. 13.
    Hong, J.-H., Cho, S.-B.: The Classification of Cancer based on DNA Microarray Data That Uses Diverse Ensemble Genetic Programming. Artificial Intelligence in Medicine. 36(1), 43–58 (2006)CrossRefGoogle Scholar
  14. 14.
    Maltoni, D., Maio, D., Jain, A.K., Prabhakar, S.: Handbook of Fingerprint Recognition. Springer, Heidelberg (2003)zbMATHGoogle Scholar
  15. 15.
    Yager, N., Amin, A.: Fingerprint Classification: A Review. Pattern Analysis and Application. 7(1), 77–93 (2004)CrossRefMathSciNetGoogle Scholar
  16. 16.
    Liu, J., Li, B., Dillon, T.: An Improved Naïve Bayesian Classifier Technique Coupled with a Novel Input Solution Method. IEEE Trans. Systems, Man, and Cybernetics-Part C: Applications and Reviews 31(2), 249–256 (2001)CrossRefGoogle Scholar
  17. 17.
    Zhang, Q., Yan, H.: Fingerprint Classification based on Extraction and Analysis of Singularities and Pseudo Ridges. Pattern Recognition. 37(11), 2233–2243 (2004)CrossRefGoogle Scholar
  18. 18.
    Kuncheva, L.I., Bezdek, J.C., Duin, R.P.W.: Decision Templates for Multiple Classifier Fusion: An Experimental Comparison. Pattern Recognition. 34(2), 299–314 (2001)zbMATHCrossRefGoogle Scholar
  19. 19.
    Jain, A.K., Dubes, R.C.: Algorithms for Clustering Data. Prentice-Hall, Englewood Cliffs (1988)zbMATHGoogle Scholar
  20. 20.
    Deutsch, J.: Evolutionary Algorithms for Finding Optimal Gene Sets in Microarray Prediction. Bioinformatics. 19(1), 45–52 (2003)CrossRefGoogle Scholar
  21. 21.
    Li, T., Zhang, C., Ogihara, M.: A Comparative Study of Feature Selection and Multiclass Classification Methods for Tissue Classification based on Gene Expression. Bioinformatics. 20(15), 2429–2437 (2004)CrossRefGoogle Scholar
  22. 22.
    Yao, Y., Marcialis, G.L., Pontil, M., Frasconi, P., Roli, F.: Combining Flat and Structured Representations for Fingerprint Classification with Recursive Neural Networks and Support Vector Machines. Pattern Recognition. 36(2), 397–406 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Jun-Ki Min
    • 1
  • Jin-Hyuk Hong
    • 1
  • Sung-Bae Cho
    • 1
  1. 1.Department of Computer Science, Yonsei University, Biometrics Engineering Research Center, 134 Shinchon-dong, Sudaemoon-ku, Seoul 120-749Korea

Personalised recommendations