Evaluation of Distance Measures for Multi-class Classification in Binary SVM Decision Tree
Multi-class classification can often be constructed as a generalization of binary classification. The approach that we use for solving this kind of classification problem is SVM based Binary Decision Tree architecture (SVM-BDT). It takes advantage of both the efficient computation of the decision tree architecture and the high classification accuracy of SVMs. The hierarchy of binary decision subtasks using SVMs is designed with a clustering algorithm. In this work, we are investigating how different distance measures for the clustering influence the predictive performance of the SVM-BDT. The distance measures that we consider include Euclidian distance, Standardized Euclidean distance and Mahalanobis distance. We use five different datasets to evaluate the performance of the SVM based Binary Decision Tree architecture with different distances. Also, the performance of this architecture is compared with four other SVM based approaches, ensembles of decision trees and neural network. The results from the experiments suggest that the performance of the architecture significantly varies depending of applied distance measure in the clustering process.
KeywordsSupport Vector Machines Binary tree architecture Euclidian distance Standardized Euclidean distance and Mahalanobis distance
Unable to display preview. Download preview PDF.
- 1.Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1999)Google Scholar
- 3.Joachims, T.: Making large scale SVM learning practical. In: Scholkopf, B., Bruges, C., Smola, A. (eds.) Advances in kernel methods-support vector learning. MIT Press, Cambridge (1998)Google Scholar
- 5.Mahalanobis, P.: On tests and measures of group divergence I. Theoretical formulae, J. and Proc. Asiat. Soc. of Bengal 26, 541–588 (1930)Google Scholar
- 7.Friedman, J.H.: Another approach to polychotomous classification. Technical report, Department of Statistics, Stanford University (1997)Google Scholar
- 8.Xu, P., Chan, A.K.: Support vector machine for multi-class signal classification with unbalanced samples. In: Proceedings of the IJCNN 2003, Portland, pp. 1116–1119 (2003)Google Scholar
- 9.Platt, J., Cristianini, N., Shawe-Taylor, J.: Large margin DAGSVMs for multiclass classification. Advances in Neural Information Processing Sys. 12, 547–553 (2000)Google Scholar
- 10.Fei, B., Liu, J.: Binary Tree of SVM: A New Fast Multiclass Training and Classification Algorithm. IEEE Transaction on neural net. 17(3) (May 2006)Google Scholar
- 13.Collobert, R., Bengio, S., Mariethoz, J.: Torch: a modular machine learning software library, Technical Report IDIAP-RR 02-46, IDIAP (2002)Google Scholar
- 14.MNIST, MiniNIST, USA, http://yann.lecun.com/exdb/mnist
- 15.Gorgevik, D., Cakmakov, D.: An Efficient Three-Stage Classifier for Handwritten Digit Recognition. In: Proceedings of 17th ICPR 2004, August 23-26, vol. 4, pp. 507–510. IEEE Computer Society, Cambridge (2004)Google Scholar
- 16.Blake, C., Keogh, E., Merz, C.: UCI Repository of Machine Learning Databases (1998), http://archive.ics.uci.edu/ml/datasets.html
- 19.Martinez, J.M. (ed.) MPEG Requirements Group, ISO/MPEG N4674, Overview of the MPEG-7 Standard, v 6.0, Jeju (March 2002)Google Scholar