Towards More Efficient Multi-label Classification Using Dependent and Independent Dual Space Reduction

  • Eakasit Pacharawongsakda
  • Thanaruk Theeramunkong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7302)


While multi-label classification can be widely applied for problems where multiple classes can be assigned to an object, its effectiveness may be sacrificed due to curse of dimensionality in the feature space and sparseness of dimensionality in the label space. Moreover, it suffers with high computational cost when there exist a high number of dimensions, as well as with lower accuracy when there are a number of noisy examples. As a solution, this paper presents two alternative methods, namely Dependent Dual Space Reduction and Independent Dual Space Reduction, to reduce dimensions in the dual spaces, i.e., the feature and label spaces, using Singular Value Decomposition (SVD). The first approach constructs the covariance matrix to represent dependency between the features and labels, project both of them into a single reduced space, and then perform prediction on the reduced space. On the other hand, the second approach handles the feature space and the label space separately by constructing a covariance matrix for each space to represent feature dependency and label dependency before performing SVD on dependency profile of each space to reduce dimension and for noise elimination and then predicting using their reduced dimensions. A number of experiments evidence that prediction on the reduced spaces for both dependent and independent reduction approaches can obtain better classification performance as well as faster computation, compared to the prediction using the original spaces. The dependent approach helps saving computational time while the independent approach tends to obtain better classification performance.


multi-label classification Singular Value Decomposition SVD dimensionality reduction Problem Transformation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bi, W., Kwok, J.: Multi-label classification on tree- and dag-structured hierarchies. In: Getoor, L., Scheffer, T. (eds.) Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 17–24. ACM, New York (2011)Google Scholar
  2. 2.
    Boutell, M., Luo, J., Shen, X., Brown, C.: Learning multi-label scene classification. Pattern Recognition 37(9), 1757–1771 (2004)CrossRefGoogle Scholar
  3. 3.
    Cherman, E.A., Metz, J., Monard, M.C.: Incorporating label dependency into the binary relevance framework for multi-label classification. Expert Systems with Applications (2011)Google Scholar
  4. 4.
    Elisseeff, A., Weston, J.: A kernel method for multi-labelled classification. In: Proceedings of the Advances in Neural Information Processing Systems, vol. 14, pp. 681–687. MIT Press (2001)Google Scholar
  5. 5.
    Golub, G., Reinsch, C.: Singular value decomposition and least squares solutions. Numerische Mathematik 14, 403–420 (1970)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Han, Y., Wu, F., Jia, J., Zhuang, Y., Yu, B.: Multi-task sparse discriminant analysis (mtsda) with overlapping categories. In: Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, pp. 469–474 (2010)Google Scholar
  7. 7.
    Hsu, D., Kakade, S., Langford, J., Zhang, T.: Multi-label prediction via compressed sensing. In: Proceedings of the Advances in Neural Information Processing Systems, vol. 22, pp. 772–780 (2009)Google Scholar
  8. 8.
    Katakis, I., Tsoumakas, G., Vlahavas, I.: Multilabel text classification for automated tag suggestion. In: Proceedings of the the ECML/PKDD 2008 Discovery Challenge (2008)Google Scholar
  9. 9.
    Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier chains for multi-label classification. Machine Learning, 1–27 (2011)Google Scholar
  10. 10.
    Schapire, R., Singer, Y.: Boostexter: A boosting-based system for text categorization. Machine Learning 39(2/3), 135–168 (2000)zbMATHCrossRefGoogle Scholar
  11. 11.
    Tai, F., Lin, H.T.: Multi-label classification with principle label space transformation. In: Proceedings of the 2nd International Workshop on Learning from Multi-Label Data (MLD 2010), pp. 45–52 (2010)Google Scholar
  12. 12.
    Trohidis, K., Tsoumakas, G., Kalliris, G., Vlahavas, I.: Multi-label classification of music into emotions. In: Proceedings of the International Symposium/Conference on Music Information Retrieval, pp. 325–330 (2008)Google Scholar
  13. 13.
    Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining Multi-label Data. In: Data Mining and Knowledge Discovery Handbook, 2nd edn. Springer (2010)Google Scholar
  14. 14.
    Tsoumakas, G., Katakis, I., Vlahavas, I.: Random k-labelsets for multilabel classification. IEEE Transactions on Knowledge and Data Engineering 23, 1079–1089 (2011)CrossRefGoogle Scholar
  15. 15.
    Wang, H., Ding, C., Huang, H.: Multi-label Linear Discriminant Analysis. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6316, pp. 126–139. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  16. 16.
    Yu, K., Yu, S., Tresp, V.: Multi-label informed latent semantic indexing. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 258–265 (2005)Google Scholar
  17. 17.
    Zhang, M.L., Pea, J.M., Robles, V.: Feature selection for multi-label naive bayes classification. Information Science, 3218–3229 (2009)Google Scholar
  18. 18.
    Zhang, Y., Zhou, Z.H.: Multilabel dimensionality reduction via dependence maximization. ACM Transactions on Knowledge Discovery from Data (TKDD) 4(3), 1–21 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Eakasit Pacharawongsakda
    • 1
  • Thanaruk Theeramunkong
    • 1
  1. 1.School of Information, Computer, and Communication TechnologySirindhorn International Institute of Technology, Thammasat UniversityThailand

Personalised recommendations