Neural Processing Letters

, Volume 50, Issue 2, pp 1361–1380 | Cite as

Calibrated Multi-label Classification with Label Correlations

  • Zhi-Fen He
  • Ming YangEmail author
  • Hui-Dong Liu
  • Lei Wang


Multi-label classification is a special learning task where each instance may be associated with multiple labels simultaneously. There are two main challenges: (a) discovering and exploiting the label correlations automatically, and (b) separating the relevant labels from the irrelevant labels of each instance effectively. Nevertheless, many existing multi-label classification algorithms fail to deal with both challenges at the same time. In this paper, we integrate multi-label classification, label correlations and threshold calibration into a unified learning framework, and propose calibrated multi-label classification with label correlations, named CMLLC. Specifically, we firstly introduce a label covariance matrix to characterize the label correlations and a virtual label to calibrate label decision threshold of each instance. Secondly, the framework of our CMLLC model is constructed for joint learning of the label correlations and model parameters corresponding to each label and the virtual label. Lastly, the optimization problem is jointly convex and solved by an alternating iterative method. Experimental results on sixteen multi-label benchmark datasets in terms of five evaluation criteria demonstrate that CMLLC outperforms the state-of-the-art multi-label classification algorithms.


Multi-label classification Label correlations Threshold calibration 



This work was supported by National Natural Science Foundation of China under Grants 61876087, 61502058, the State Key Program of National Natural Science Foundation of China under Grant 61432008, the Science and Technology Research Project of Jiangxi Provincial Education Department under Grant GJJ151262, Natural Science Foundation of Educational Committee of Jiangsu Province under Grant 15KJB520002, and the Social Science Research Project of Pingxiang under Grant 2017XW02. The authors would like to thank the anonymous reviewers and the editors for their helpful comments and suggestions.


  1. 1.
    Schapire RE, Singer Y (2000) Boostexter: a boosting-based system for text categorization. Mach Learn 39(2/3):135–168CrossRefGoogle Scholar
  2. 2.
    Zhang ML, Zhou ZH (2006) Multilabel neural networks with applications to functional genomics and text categorization. IEEE Trans Knowl Data Eng 18(10):1338–1351CrossRefGoogle Scholar
  3. 3.
    Yan Y, Wang Y, Gao WC (2018) LSTM: multi-label ranking for document classification. Neural Process Lett 47(1):117–138CrossRefGoogle Scholar
  4. 4.
    Boutell MR, Luo J, Luo JB, Shen XP, Brown CM (2004) Learning multi-label scene classification. Pattern Recognit 37(9):1757–1771CrossRefGoogle Scholar
  5. 5.
    Jiang A, Wang C, Zhu Y (2008) Calibrated rank-SVM for multi-label image categorization. In: Proceedings of the international joint conference on neural networks, Hong Kong, China, pp 1450–1455Google Scholar
  6. 6.
    Liu W, Yang X, Tao D (2018) Multiview dimension reduction via Hessian multiset canonical correlations. Inf Fusion 41:119–128CrossRefGoogle Scholar
  7. 7.
    Yu J, Zhang B, Kuang Z (2017) iPrivacy: image privacy protection by identifying sensitive objects via deep multi-task learning. IEEE Trans Inf Forensics Secur 12(5):1005–1016CrossRefGoogle Scholar
  8. 8.
    Yu J, Yang X, Gao F (2017) Deep multimodal distance metric learning using click constraints for image ranking. IEEE Trans Cybern 47(12):4014–4024CrossRefGoogle Scholar
  9. 9.
    Tao D, Hong C, Yu J (2015) Multimodal deep autoencoder for human pose recovery. IEEE Trans Image Process 24(12):5659–5670MathSciNetCrossRefGoogle Scholar
  10. 10.
    Trohidis K, Tsoumakas G, Kalliris G, Vlahavas IP (2008) Multilabel classification of music into emotions. In: Proceedings of the 9th international conference on music information retrieval, Philadephia, PA, USA, pp 325–330Google Scholar
  11. 11.
    Zhang ML, Zhou ZH (2014) A review on multi-label learning algorithms. IEEE Trans Knowl Data Eng 26(8):1819–1837CrossRefGoogle Scholar
  12. 12.
    Huang SJ, Yu Y, Zhou ZH (2012) Multi-label hypothesis reuse. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining, Beijing, China, pp 525–533Google Scholar
  13. 13.
    Elisseeff A, Weston J (2001) A kernel method for multi-labelled classification. In: Proceedings of the 14th conference on neural information processing systems (NIPS2001), Vancouver, British Columbia, Canada, pp 681–687Google Scholar
  14. 14.
    Zhang ML, Pena JM, Robles V (2009) Feature selection for multi-label naive Bayes classification. Inf Sci 179(19):3218–3229CrossRefGoogle Scholar
  15. 15.
    Zhang ML (2009) ML-RBF: RBF neural networks for multi-label learning. Neural Process Lett 29(2):61–74CrossRefGoogle Scholar
  16. 16.
    Read J, Pfahringer B, Holmes G, Frank E (2011) Classifier chains for multi-label classification. Mach Learn 85(3):333–359Google Scholar
  17. 17.
    Tsoumakas G, Katakis I, Vlahavas I (2008) Effective and efficient multilabel classification in domains with large number of labels. In: Proceedings of ECML/PKDD 2008 workshop on mining multidimensional data, Antwerp, Belgium, pp 30–44Google Scholar
  18. 18.
    Ghamrawi N, Mccallum A (2005) Collective multilabel classification. In: Proceedings of the 14th ACM international conference on information and knowledge management, Bremen, Germany, pp 195–200Google Scholar
  19. 19.
    Godbole S, Sarawagi S (2004) Discriminative methods for multi-labeled classification. In: Dai H, Srikant R, Zhang C (eds) Lecture Notes in Artificial Intelligence, vol 3056. Springer, Berlin, pp 22–30Google Scholar
  20. 20.
    Chen G, Song YQ, Wang F, et al (2008) Semi-supervised multi-label learning by solving a Sylvester equation. In: SIAM conference on data mining, Atlanta, Georgia, pp 410–419Google Scholar
  21. 21.
    Gu Q, Li Z, Han J (2011) Correlated multi-label feature selection. In: Proceedings of the 20th ACM international conference on information and knowledge management, Glasgow, Scotland, UK, pp 1087–1096Google Scholar
  22. 22.
    Zhang Y, Yeung DY (2013) Multilabel relationship learning. ACM Trans Knowl Discov Data 7(2):1–30CrossRefGoogle Scholar
  23. 23.
    Zhu Y, Kwok JT, Zhou ZH (2017) Multi-Label Learning with Global and Local Label Correlation. IEEE Trans Knowl Data Eng, arXiv preprint, arXiv:1704:01415
  24. 24.
    He ZF, Yang M, Liu HD (2014) Joint learning of multi-label classification and label correlations. J Softw 25(9):1967–1981 (in Chinese)zbMATHGoogle Scholar
  25. 25.
    Hullermeier E, Furnkranz J, Cheng W, Brinker K (2008) Label ranking by learning pairwise preferences. Artif Intell 172(16):1897–1916MathSciNetCrossRefGoogle Scholar
  26. 26.
    Furnkranz F, Hullermeier E, Mencia EL, Brinker K (2008) Multilabel classification via calibrated label ranking. Mach Learn 73(2):133–153CrossRefGoogle Scholar
  27. 27.
    Read J (2008) A pruned problem transformation method for multi-label classification. In: Proceedings of New Zealand computer science research student conference, Christchurch, New Zealand, pp 143–150Google Scholar
  28. 28.
    Tsoumakas G, Katakis I, Vlahavas I (2011) Random k-labelsets for multilabel classification. IEEE Trans Knowl Data Eng 23(7):1079–1089CrossRefGoogle Scholar
  29. 29.
    Gharroudi O, Elghazel H, Aussem A (2015) Calibrated k-labelsets for ensemble multi-label classification. In: Proceedings of international conference on neural information processing, pp 573–582Google Scholar
  30. 30.
    He ZF, Yang M, Liu HD (2015) Multi-task joint feature selection for multi-label classification. Chin J Electron 24(CJE–2):281–287CrossRefGoogle Scholar
  31. 31.
    Sun Z, Zhao Y, Cao D, Hao H (2017) Hierarchical multilabel classification with optimal path prediction. Neural Process Lett 45(1):263–277CrossRefGoogle Scholar
  32. 32.
    Xu J (2012) An efficient multi-label support vector machine with a zero label. Expert Syst Appl 39(5):4796–4804CrossRefGoogle Scholar
  33. 33.
    Xu J (2014) Multi-label core vector machine with a zero label. Pattern Recognit 47(7):2542–2557CrossRefGoogle Scholar
  34. 34.
    Clare A, King RD (2001) Knowledge discovery in multi-label phenotypedata. In: Raedt LD, Siebes A (eds) Lecture Notes in Computer Science. Springer, Berlin, pp 42–53zbMATHGoogle Scholar
  35. 35.
    Zhang ML, Zhou ZH (2007) ML-KNN: a lazy learning approach to multi-label learning. Pattern Recognit 40(7):2038–2048CrossRefGoogle Scholar
  36. 36.
    Kwok JT (1999) Moderating the outputs of support vector machine classifiers. IEEE Trans Neural Netw 10(5):1018–1031CrossRefGoogle Scholar
  37. 37.
    Xu J (2013) Fast multi-label core vector machine. Pattern Recognit 46(3):885–898CrossRefGoogle Scholar
  38. 38.
    Zhang Y, Yeung DY (2010) A convex formulation for learning task relationships in multi-task learning. In: Proceedings of the 26th conference on uncertainty in artificial intelligence, Catalina Island, California, pp 733–742Google Scholar
  39. 39.
    Chao G, Sun S (2016) Consensus and complementarity based maximum entropy discrimination for multi-view classification. Inf Sci 367–368:296–310CrossRefGoogle Scholar
  40. 40.
    Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, New YorkCrossRefGoogle Scholar
  41. 41.
    Chen J, Ye J (2008) Training SVM with indefinite kernels. In: Proceedings of the 25th international conference on machine learning, Helsinki, Finland, pp 136–143Google Scholar
  42. 42.
    Tsoumakas G, Xioufis ES, Vilcek J (2011) Mulan: a java library for multi-label learning. J Mach Learn Res 12(7):2411–2414MathSciNetzbMATHGoogle Scholar
  43. 43.
    Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Zhi-Fen He
    • 1
    • 2
  • Ming Yang
    • 1
    Email author
  • Hui-Dong Liu
    • 3
  • Lei Wang
    • 4
  1. 1.School of Computer Science and TechnologyNanjing Normal UniversityNanjingPeople’s Republic of China
  2. 2.School of Information and Computer EngineeringPingxiang UniversityPingxiangPeople’s Republic of China
  3. 3.Department of Computer ScienceStony Brook UniversityStony BrookUSA
  4. 4.School of Computing and Information TechnologyUniversity of WollongongWollongongAustralia

Personalised recommendations