Machine Learning

, Volume 76, Issue 2–3, pp 211–225 | Cite as

Combining instance-based learning and logistic regression for multilabel classification



Multilabel classification is an extension of conventional classification in which a single instance can be associated with multiple labels. Recent research has shown that, just like for conventional classification, instance-based learning algorithms relying on the nearest neighbor estimation principle can be used quite successfully in this context. However, since hitherto existing algorithms do not take correlations and interdependencies between labels into account, their potential has not yet been fully exploited. In this paper, we propose a new approach to multilabel classification, which is based on a framework that unifies instance-based learning and logistic regression, comprising both methods as special cases. This approach allows one to capture interdependencies between labels and, moreover, to combine model-based and similarity-based inference for multilabel classification. As will be shown by experimental studies, our approach is able to improve predictive accuracy in terms of several evaluation criteria for multilabel prediction.


Multilabel classification Instance-based learning Nearest neighbor classification Logistic regression Bayesian inference 


  1. Aha, D., Kibler, D., & Alber, M. (1991). Instance-based learning algorithms. Machine Learning, 6(1), 37–66. Google Scholar
  2. Boutell, M. R., Luo, J., Shen, X., & Brown, C. M. (2004). Learning multilabel scene classification. Pattern Recognition, 37(9), 1757–1771. CrossRefGoogle Scholar
  3. Clare, A., & King, R. D. (2001). Knowledge discovery in multilabel phenotype data. In L. D. Raedt & A. Siebes (Eds.), Lecture notes in computer science (Vol. 2168, pp. 42–53). Berlin: Springer. Google Scholar
  4. Comite, F. D., Gilleron, R., & Tommasi, M. (2003). Learning multilabel alternating decision tree from texts and data. In P. Perner & A. Rosenfeld (Eds.), Lecture notes in computer science (Vol. 2734, pp. 35–49). Berlin: Springer. Google Scholar
  5. Dasarathy, B. V., editor (1991). Nearest neighbor (NN) norms: NN pattern classification techniques. Los Alamitos: IEEE Comput. Soc. Google Scholar
  6. Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30. MathSciNetGoogle Scholar
  7. Elisseeff, A., & Weston, J. (2002). A kernel method for multilabelled classification. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems (Vol. 14, pp. 681–687). Cambridge: MIT Press. Google Scholar
  8. Getoor, L., & Taskar, B., editors (2007). Introduction to statistical relational learning. Cambridge: MIT Press. MATHGoogle Scholar
  9. Ghamrawi, N., & McCallum, A. (2005). Collective multilabel classification. In Proc. CIKM-05, Bremen, Germany. Google Scholar
  10. Godbole, S., & Sarawagi, S. (2004). Discriminative methods for multilabeled classification. In LNCS: Vol. 3056. Advances in knowledge discovery and data mining (pp. 20–33). Berlin: Springer. Google Scholar
  11. Kazawa, H., Izumitani, T., Taira, H., & Maeda, E. (2005). Maximal margin labeling for multi-topic text categorization. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural inf. proc. syst. (Vol. 17). Cambridge: MIT Press. Google Scholar
  12. Lu, Q., & Getoor, L. (2003). Link-based classification. In Proc. ICML-03 (pp. 496–503) Washington. Google Scholar
  13. Maron, O., & Ratan, A. L. (1998). Multiple-instance learning for natural scene classification. In Proc. ICML (pp. 341–349), Madison, WI. Google Scholar
  14. Schapire, R. E., & Singer, Y. (2000). Boostexter: a boosting-based system for text categorization. Machine Learning, 39(2), 135–168. MATHCrossRefGoogle Scholar
  15. Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1–47. CrossRefGoogle Scholar
  16. Snoek, C. G. M., Worring, M., van Gemert, J. C., Geusebroek, J. M., & Smeulders, A. W. M. (2006). The challenge problem for automated detection of 101 semantic concepts in multimedia. In Proc. ACM multimedia (pp. 421–430), Santa Barbara, USA. Google Scholar
  17. Trohidis, K., Tsoumakas, G., Kalliris, G., & Vlahavas, I. (2008). Multilabel classification of music into emotions. In Proc. int. conf. music information retrieval. Google Scholar
  18. Tsoumakas, G., & Katakis, I. (2007). Multi-label classification: An overview. International Journal of Data Warehousing and Mining, 3(3), 1–13. Google Scholar
  19. Ueda, N., & Saito, K. (2003). Parametric mixture models for multilabel text. In S. Becker & S. Thrun (Eds.), Advances in neural information processing (Vol. 15, pp. 721–728). Cambridge: MIT Press. Google Scholar
  20. Vens, C., Struyf, J., Schietgat, L., Dzeroski, S., & Blockeel, H. (2008). Decision trees for hierarchical multilabel classification. Machine Learning, 73, 185–214. CrossRefGoogle Scholar
  21. Witten, I., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). San Francisco: Morgan Kaufmann. MATHGoogle Scholar
  22. Zhang, M.-L., & Zhou, Z.-H. (2006). Multi-label neural networks with applications to functional genomics and text categorization. In IEEE transactions on knowledge and data engineering (Vol. 18, pp. 1338–1351). Google Scholar
  23. Zhang, M.-L., & Zhou, Z.-H. (2007). ML-kNN: A lazy learning approach to multilabel learning. Pattern Recognition, 40(7), 2038–2048. MATHCrossRefGoogle Scholar
  24. Zhou, Z.-H., & Zhang, M.-L. (2007). Multi-instance multilabel learning with application to scene classification. In B. Schölkopf, J. Platt, & T. Hofmann (Eds.), Advances in neural inf. proc. syst. (Vol. 19, pp. 1609–1616). Cambridge: MIT Press. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Department of Mathematics and Computer ScienceUniversity of MarburgMarburgGermany

Personalised recommendations