An Empirical Comparison of Hierarchical vs. Two-Level Approaches to Multiclass Problems
The Error Correcting Output Codes (ECOC) framework provides a powerful and popular method for solving multiclass problems using a multitude of binary classifiers. We had recently introduced  the Binary Hierarchical Classifier (BHC) architecture that addresses multiclass classification problems using a set of binary classifiers organized in the form of a hierarchy. Unlike ECOCs, the BHC groups classes according to their natural affinities in order to make each binary problem easier. However, it cannot exploit the powerful error correcting properties of an ECOC ensemble, which can provide good results even when the individual classifiers are weak. In this paper, we provide an empirical comparison of these two approaches on a variety of datasets, using well-tuned SVMs as the base classifiers. The results show that while there is no clear advantage to either technique in terms of classification accuracy, BHCs typically achieve this performance using fewer classifiers, and have the added advantage of automatically generating a hierarchy of classes. Such hierarchies often provide a valuable tool for extracting domain knowledge, and achieve better results when coarser granularity of the output space is acceptable.
KeywordsBinary Classifier Output Space Bayesian Classifier Empirical Comparison Code Matrix
Unable to display preview. Download preview PDF.
- 5.Hastie, T., Tibshirani, R.: Classification by Pairwise Coupling. In: Hastie, T., Tibshirani, R. (eds.) Advances in Neural Information Processing Systems, vol. 10, The MIT Press, Cambridge (1998)Google Scholar
- 7.Allwein, E.L., Schapire, R.E., Singer, Y.: Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers. In: Proc. 17th International Conf. on Machine Learning, pp. 9–16. Morgan Kaufmann, San Francisco (2000)Google Scholar
- 8.Crammer, K., Singer, Y.: On the Learnability and Design of Output Codes for Multiclass Problems. Computational Learning Theory, 35–46 (2000)Google Scholar
- 11.Morgan, T.J., Henneguelle, A., Ham, J., Ghosh, J., Crawford, M.M.: Adaptive Feature Spaces for Land Cover Classification with Limited Ground Truth Data. Kittler, J., Roli, F. (eds.) International Journal of Pattern Recognition and Artificial Intelligence (2004) (to appear)Google Scholar
- 12.Kumar, S., Ghosh, J.: GAMLS: A Generalized framework for Associative Modular Learning Systems. In: Application and Science of Computational Intelligence II, SPIE, vol. 3722, pp. 24–35 (1999)Google Scholar
- 15.Kong, E.B., Dietterich, T.G.: Error-Correcting Output Coding Corrects Bias and Variance. In: International Conference on Machine Learning, pp. 313–321 (1995)Google Scholar
- 16.Bose, R.C., Ray-Chauduri, D.K.: On a Class of Error Correcting Binary Group Codes. Information and Control (3), 68–79 (1960)Google Scholar
- 18.Blake, C.L., Merz, C.J.: UCI Repository of Machine Learning Databases, University of California, Department of Information and Computer Science, Irvine, CA (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html