An Empirical Comparison of Flat and Hierarchical Performance Measures for Multi-Label Classification with Hierarchy Extraction

  • Florian Brucker
  • Fernando Benites
  • Elena Sapozhnikova
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6881)


Multi-label Classification (MC) often deals with hierarchically organized class taxonomies. In contrast to Hierarchical Multi-label Classification (HMC), where the class hierarchy is assumed to be known a priori, we are interested in the opposite case where it is unknown and should be extracted from multi-label data automatically. In this case the predictive performance of a classifier can be assessed by well-known Performance Measures (PMs) used in flat MC such as precision and recall. The fact that these PMs treat all class labels as independent labels, in contrast to hierarchically structured taxonomies, is a problem. As an alternative, special hierarchical PMs can be used that utilize hierarchy knowledge and apply this knowledge to the extracted hierarchy. This type of hierarchical PM has only recently been mentioned in literature. The aim of this study is first to verify whether HMC measures do significantly improve quality assessment in this setting. In addition, we seek to find a proper measure that reflects the potential quality of extracted hierarchies in the best possible way. We empirically compare ten hierarchical and four traditional flat PMs in order to investigate relations between them. The performance measurements obtained for predictions of four multi-label classifiers ML-ARAM, ML-kNN, BoosTexter and SVM on four datasets from the text mining domain are analyzed by means of hierarchical clustering and by calculating pairwise statistical consistency and discriminancy.


Multi-label Classification Text Classification Performance Measures Hierarchical Classification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Benites, F., Brucker, F., Sapozhnikova, E.: Multi-Label Classification by ART-based Neural Networks and Hierarchy Extraction. In: Proc. of the IEEE IJCNN 2010, pp. 2788–2796. IEEE Computer Society, Barcelona (2010)Google Scholar
  2. 2.
    Brucker, F., Benites, F., Sapozhnikova, E.: Multi-label classification and extracting predicted class hierarchies. Pattern Recognition 44(3), 724–738 (2011)CrossRefMATHGoogle Scholar
  3. 3.
    Cai, L., Hofmann, T.: Exploiting known taxonomies in learning overlapping concepts. In: Proc. of Int. Joint Conf. on Artificial Intelligence (2007)Google Scholar
  4. 4.
    Cesa-Bianchi, N., Gentile, C., Zaniboni, L.: Hierarchical classification: combining Bayes with SVM. In: Proc. of the 23rd Int. Conf. on Machine learning (2006)Google Scholar
  5. 5.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), (acc. 03.2010)
  6. 6.
    Costa, E., Lorena, A., Carvalho, A., Freitas, A.: A review of performance evaluation measures for hierarchical classifiers. In: Proc. of the AAAI 2007 Workshop: Evaluation Methods for Machine Learning II, pp. 1–6 (2007)Google Scholar
  7. 7.
    Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proc. of the 23rd Int. Conf. on Machine Learning, p. 240. ACM, New York (2006)Google Scholar
  8. 8.
    Granitzer, M.: Hierarchical text classification using methods from machine learning. Master’s thesis, Graz University of Technology (2003)Google Scholar
  9. 9.
    Huang, J., Ling, C.: Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 299–310 (2005)Google Scholar
  10. 10.
    Ipeirotis, P., Gravano, L., Sahami, M.: Probe, Count, and Classify: Categorizing Hidden-Web Databases. In: Proc. of the 2001 ACM SIGMOD Int. Conf. on Management of Data, pp. 67–78 (2001)Google Scholar
  11. 11.
    Kiritchenko, S.: Hierarchical text categorization and its application to bioinformatics. Ph.D. thesis, University of Ottawa Ottawa, Ont., Canada (2006)Google Scholar
  12. 12.
    Nowak, S., Lukashevich, H.: Multilabel classification evaluation using ontology information. In: Proc. of the First ESWC Workshop on Inductive Reasoning and Machine Learning on the Semantic Web, Heraklion, Greece (2009)Google Scholar
  13. 13.
    Silla, C., Freitas, A.: A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 1–42 (2010)Google Scholar
  14. 14.
    Struyf, J., Dzeroski, S., Blockeel, H., Clare, A.: Hierarchical multi-classification with predictive clustering trees in functional genomics. In: Bento, C., Cardoso, A., Dias, G. (eds.) EPIA 2005. LNCS (LNAI), vol. 3808, pp. 272–283. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Sun, A., Lim, E.: Hierarchical text classification and evaluation. In: Proc. of the 2001 IEEE Int. Conf. on Data Mining, California, USA, vol. 528 (2001)Google Scholar
  16. 16.
    Tan, P., Steinbach, M., Kumar, V.: Introduction to data mining. Pearson Addison Wesley, Boston (2006)Google Scholar
  17. 17.
    Verspoor, K., Cohn, J., Mniszewski, S., Joslyn, C.: A categorization approach to automated ontological function annotation. Protein Science 15(6), 1544–1549 (2006)CrossRefGoogle Scholar
  18. 18.
    Wang, K., Zhou, S., He, Y.: Hierarchical classification of real life documents. In: Proc. of the 1st (SIAM) Int. Conf. on Data Mining, pp. 1–16 (2001)Google Scholar
  19. 19.
    Woolam, C., Khan, L.: Multi-concept document classification using a perceptron-like algorithm. In: WI-IAT 2008: Proc. of the 2008 IEEE/WIC/ACM Int. Conf. on Web Intelligence and Intelligent Agent Technology, pp. 570–574. IEEE Computer Society, Washington, DC, USA (2008)CrossRefGoogle Scholar
  20. 20.
    Wu, F., Zhang, J., Honavar, V.: Learning Classifiers Using Hierarchically Structured Class Taxonomies. In: Zucker, J.-D., Saitta, L. (eds.) SARA 2005. LNCS (LNAI), vol. 3607, pp. 313–320. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  21. 21.
    Yang, Y.: An evaluation of statistical approaches to text categorization. Information Retrieval 1(1), 69–90 (1999)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Florian Brucker
    • 1
  • Fernando Benites
    • 1
  • Elena Sapozhnikova
    • 1
  1. 1.Department of Computer and Information ScienceUniversity of KonstanzGermany

Personalised recommendations