Date: 08 Aug 2012
Exploiting label dependencies for improved sample complexity
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Multi-label classification exhibits several challenges not present in the binary case. The labels may be interdependent, so that the presence of a certain label affects the probability of other labels’ presence. Thus, exploiting dependencies among the labels could be beneficial for the classifier’s predictive performance. Surprisingly, only a few of the existing algorithms address this issue directly by identifying dependent labels explicitly from the dataset. In this paper we propose new approaches for identifying and modeling existing dependencies between labels. One principal contribution of this work is a theoretical confirmation of the reduction in sample complexity that is gained from unconditional dependence. Additionally, we develop methods for identifying conditionally and unconditionally dependent label pairs; clustering them into several mutually exclusive subsets; and finally, performing multi-label classification incorporating the discovered dependencies. We compare these two notions of label dependence (conditional and unconditional) and evaluate their performance on various benchmark and artificial datasets. We also compare and analyze labels identified as dependent by each of the methods. Moreover, we define an ensemble framework for the new methods and compare it to existing ensemble methods. An empirical comparison of the new approaches to existing base-line and state-of-the-art methods on 12 various benchmark datasets demonstrates that in many cases the proposed single-classifier and ensemble methods outperform many multi-label classification algorithms. Perhaps surprisingly, we discover that the weaker notion of unconditional dependence plays the decisive role.
Editors: Grigorios Tsoumakas, Min-Ling Zhang, and Zhi-Hua Zhou.
Baum, E. B., & Haussler, D. (1989). What size net gives valid generalization? Neural Computation, 1(1), 151–160. CrossRef
Dembczynski, K., Cheng, W., & Hullermeier, E. (2010a). Bayes optimal multilabel classification via probabilistic classifier chains. In Proc. ICML 2010, Haifa, Israel.
Dembczynski, K., Waegeman, W., Cheng, W., & Hüllermeier, E. (2010b). On label dependence in multi-label classification. Working notes of the 2nd international workshop on learning from multi-label data, Haifa, Israel.
Ghamrawi, N., & McCallum, A. (2005). Collective multi-label classification. In CIKM 2005 (pp. 195–200).
Kearns, M. J., Schapire, R. E., & Sellie, L. (1994). Toward efficient agnostic learning. Machine Learning, 17(2–3), 115–141. MATH
Read, J., Pfahringer, B., & Holmes, G. (2008). Multi-label classification using ensembles of pruned sets. In Proceedings of eighth IEEE international conference on data mining (pp. 995–1000).
Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2009). Classifier chains for multi-label classification. In Proceedings of 20th European conference on machine learning and knowledge discovery in databases (Vol. 2, pp. 254–269). CrossRef
Rokach, L. (2010). Pattern classification using ensemble methods. Series in machine perception and artificial intelligence: Vol. 75. Singapore: World Scientific. MATH
Rokach, L., & Maimon, O. (2005). Feature set decomposition for decision trees. Journal of Intelligent Data Analysis, 9(2), 131–158.
Tenenboim, L., Rokach, L., & Shapira, B. (2009). Multi-label classification by analyzing labels dependencies. In G. Tsoumakas, M. L. Zhang, & Z. H. Zhou (Eds.), Proceedings of the 1st international workshop on learning from multi-label data, Bled, Slovenia (pp. 117–132).
Tsoumakas, G., & Vlahavas, I. (2007). Random k-labelsets: an ensemble method for multilabel classification. In Proceedings of 18th European conference on machine learning, Warsaw, Poland (pp. 406–417).
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2008). Effective and efficient multilabel classification in domains with large number of labels. In Proceedings of ECML/PKDD 2008 workshop on mining multidimensional data (pp. 30–44).
Tsoumakas, G., Dimou, A., Spyromitros, E., Mezaris, V., Kompatsiaris, I., & Vlahavas, I. (2009). Correlation-based pruning of stacked binary relevance models for multi-label learning. In G. Tsoumakas, M. L. Zhang, & Z. H. Zhou (Eds.), Proceedings of the 1st international workshop on learning from multi-label data, Bled, Slovenia (pp. 101–116).
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In O. Maimon & L. Rokach (Eds.), Data mining and knowledge discovery handbook (2nd ed., pp. 667–686). New York: Springer.
Vapnik, V. N. (1995). The nature of statistical learning theory. New York: Springer. MATH
Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5, 241–259. CrossRef
Xu, J. (2010). Constructing a fast algorithm for multi-label classification with support vector data description. In IEEE international conference on granular computing (pp. 817–821).
- Exploiting label dependencies for improved sample complexity
Volume 91, Issue 1 , pp 1-42
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Multi-label classification
- Conditional and unconditional label dependence
- Generalization bounds
- Multi-label evaluation measures
- Ensemble learning algorithms
- Ensemble models diversity
- Empirical experiment
- Artificial datasets
- Industry Sectors
- Author Affiliations
- 1. Department of Information Systems Engineering and Telekom Innovation Laboratories, Ben-Gurion University of the Negev, Beer-Sheva, 84105, Israel
- 2. IBM Research, Haifa, Israel
- 3. Department of Computer Science, Ben-Gurion University of the Negev, Beer-Sheva, 84105, Israel