Evaluation of Distance Measures for Multi-class Classification in Binary SVM Decision Tree

  • Gjorgji Madzarov
  • Dejan Gjorgjevikj
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6113)

Abstract

Multi-class classification can often be constructed as a generalization of binary classification. The approach that we use for solving this kind of classification problem is SVM based Binary Decision Tree architecture (SVM-BDT). It takes advantage of both the efficient computation of the decision tree architecture and the high classification accuracy of SVMs. The hierarchy of binary decision subtasks using SVMs is designed with a clustering algorithm. In this work, we are investigating how different distance measures for the clustering influence the predictive performance of the SVM-BDT. The distance measures that we consider include Euclidian distance, Standardized Euclidean distance and Mahalanobis distance. We use five different datasets to evaluate the performance of the SVM based Binary Decision Tree architecture with different distances. Also, the performance of this architecture is compared with four other SVM based approaches, ensembles of decision trees and neural network. The results from the experiments suggest that the performance of the architecture significantly varies depending of applied distance measure in the clustering process.

Keywords

Support Vector Machines Binary tree architecture Euclidian distance Standardized Euclidean distance and Mahalanobis distance 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1999)Google Scholar
  2. 2.
    Burges, C.J.C.: A tutorial on support vector machine for pattern recognition. Data Min. Knowl. Disc. 2, 121 (1998)CrossRefGoogle Scholar
  3. 3.
    Joachims, T.: Making large scale SVM learning practical. In: Scholkopf, B., Bruges, C., Smola, A. (eds.) Advances in kernel methods-support vector learning. MIT Press, Cambridge (1998)Google Scholar
  4. 4.
    Madzarov, G., Gjorgjevikj, D., Chorbev, I.: A multi-class SVM classifier utilizing binry decision tree. An International Journal of Computing and Informatics, Informatica 33(2), 233–241 (2009)MathSciNetGoogle Scholar
  5. 5.
    Mahalanobis, P.: On tests and measures of group divergence I. Theoretical formulae, J. and Proc. Asiat. Soc. of Bengal 26, 541–588 (1930)Google Scholar
  6. 6.
    Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)MATHGoogle Scholar
  7. 7.
    Friedman, J.H.: Another approach to polychotomous classification. Technical report, Department of Statistics, Stanford University (1997)Google Scholar
  8. 8.
    Xu, P., Chan, A.K.: Support vector machine for multi-class signal classification with unbalanced samples. In: Proceedings of the IJCNN 2003, Portland, pp. 1116–1119 (2003)Google Scholar
  9. 9.
    Platt, J., Cristianini, N., Shawe-Taylor, J.: Large margin DAGSVMs for multiclass classification. Advances in Neural Information Processing Sys. 12, 547–553 (2000)Google Scholar
  10. 10.
    Fei, B., Liu, J.: Binary Tree of SVM: A New Fast Multiclass Training and Classification Algorithm. IEEE Transaction on neural net. 17(3) (May 2006)Google Scholar
  11. 11.
    Kocev, D., Vens, C., Struyf, J., Dzeroski, S.: Ensembles of multi-objective decision trees. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS (LNAI), vol. 4701, pp. 624–631. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  12. 12.
    Blockeel, H., Struyf, J.: Efficient Algorithms for Decision Tree Cross-validation. Journal of Machine Learning Research 3, 621–650 (2002)CrossRefGoogle Scholar
  13. 13.
    Collobert, R., Bengio, S., Mariethoz, J.: Torch: a modular machine learning software library, Technical Report IDIAP-RR 02-46, IDIAP (2002)Google Scholar
  14. 14.
    MNIST, MiniNIST, USA, http://yann.lecun.com/exdb/mnist
  15. 15.
    Gorgevik, D., Cakmakov, D.: An Efficient Three-Stage Classifier for Handwritten Digit Recognition. In: Proceedings of 17th ICPR 2004, August 23-26, vol. 4, pp. 507–510. IEEE Computer Society, Cambridge (2004)Google Scholar
  16. 16.
    Blake, C., Keogh, E., Merz, C.: UCI Repository of Machine Learning Databases (1998), http://archive.ics.uci.edu/ml/datasets.html
  17. 17.
  18. 18.
  19. 19.
    Martinez, J.M. (ed.) MPEG Requirements Group, ISO/MPEG N4674, Overview of the MPEG-7 Standard, v 6.0, Jeju (March 2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Gjorgji Madzarov
    • 1
  • Dejan Gjorgjevikj
    • 1
  1. 1.Department of Computer Science and EngineeringSs. Cyril and Methodius UniversityMacedonia

Personalised recommendations