Evaluation Methods of Hierarchical Models

  • Abdulqader M. AlmarsEmail author
  • Ibrahim A. IbrahimEmail author
  • Xin ZhaoEmail author
  • Sanad Al-MaskariEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11323)


In this paper, we consider the problem of evaluating the quality of hierarchical models. This task arises due to the current researchers use subjective evaluation, such as a survey to test the goodness of a hierarchy discovered by their models. We propose three methods to evaluate the quality of hierarchy extracted from unstructured text. These methods are used to reflects three important characteristics of an optimal tree: (1) Coverage which reflects a topic on a high level, close to the root node, should cover a wider range of sub-concepts than those on a lower level; (2) Parent-child relentless which means the parent topic in the tree should be semantically related to its children rather than to its non-children; (3) Topic coherence that identifies all words within a topic should be semantically related to the other words. Moreover, we introduce a new metric called, Interest-based coherent to evaluate the hierarchical tree extracted from structured data like relational data. We compare different state-of-art methods and perform extensive experiments on three real datasets. The results confirm that the proposed methods can properly evaluate the quality of the hierarchy discovered by several models.


Hierarchical models Ontology learning Evaluation methods Structured data Unstructured data 



The author would like to thank the University of Thibah for the scholarship. This research is partially supported by Nature Science Foundation of China (Grant No. 61672284) and Australian Research Council (ARC) Discovery Project DP160104075. We would like to thank Lemma solutions for their helps and valuable advice.


  1. 1.
    Ibrahim, I.A., Almars, A., Pokharel, S., Zhao, X., Li, X.: Interesting recommendations based on hierarchical visualizations of medical data (2018)Google Scholar
  2. 2.
    Almars, A., Li, X., Ibrahim, I.A., Zhao, X.: Learning concept hierarchy from short texts using context coherence. In: Hacid, H., Cellary, W., Wang, H., Paik, H.-Y., Zhou, R. (eds.) WISE 2018. LNCS, vol. 11233, pp. 319–329. Springer, Cham (2018). Scholar
  3. 3.
    Almars, A., Li, X., Zhao, X., Ibrahim, I.A., Yuan, W., Li, B.: Structured sentiment analysis. In: Cong, G., Peng, W.C., Zhang, W.E., Li, C., Sun, A. (eds.) Advanced Data Mining and Applications (2017)CrossRefGoogle Scholar
  4. 4.
    Blei, D.M., Griffiths, T.L., Jordan, M.I.: The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. J. ACM 57, 7 (2010)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Chen, H., Yin, H., Li, X., Wang, M., Chen, W., Chen, T.: People opinion topic model: opinion based user clustering in social networks. In: Proceedings of the 26th International Conference on World Wide Web Companion, WWW 2017 Companion (2017)Google Scholar
  6. 6.
    Chen, P., Zhang, N.L., Liu, T., Poon, L.K.M., Chen, Z.: Latent tree models for hierarchical topic detection. CoRR abs/1605.06650 (2016)Google Scholar
  7. 7.
    Chen, W., et al.: EEG-based motion intention recognition via multi-task RNNs, pp. 279–287. SIAM (2018)Google Scholar
  8. 8.
    Church, K.W., Hanks, P.: Word association norms, mutual information, and lexicography. Comput. Linguist. 16, 22–29 (1990)Google Scholar
  9. 9.
    Gerani, S., Carenini, G., Ng, R.T.: Modeling content and structure for abstractive review summarization. Comput. Speech Lang. (2016)Google Scholar
  10. 10.
    Ibrahim, I.A., Albarrak, A.M., Li, X.: Constrained recommendations for query visualizations. Knowl. Inf. Syst. 51, 499–529 (2017)CrossRefGoogle Scholar
  11. 11.
    Kim, J.H., Kim, D., Kim, S., Oh, A.: Modeling topic hierarchies with the recursive Chinese restaurant process. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 783–792. ACM (2012)Google Scholar
  12. 12.
    Kim, S., Zhang, J., Chen, Z., Oh, A.H., Liu, S.: A hierarchical aspect-sentiment model for online reviews. In: AAAI (2013)Google Scholar
  13. 13.
    Li, W., McCallum, A.: Pachinko allocation: dag-structured mixture models of topic correlations. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 577–584. ACM, New York (2006)Google Scholar
  14. 14.
    Mimno, D., Li, W., McCallum, A.: Mixtures of hierarchical topics with pachinko allocation. In: Proceedings of the 24th International Conference on Machine Learning, ICML 2007, pp. 633–640 (2007)Google Scholar
  15. 15.
    Mimno, D., Wallach, H.M., Talley, E., Leenders, M., McCallum, A.: Optimizing semantic coherence in topic models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, pp. 262–272. Association for Computational Linguistics, Stroudsburg (2011)Google Scholar
  16. 16.
    Wang, C., et al.: A phrase mining framework for recursive construction of a topical hierarchy. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, pp. 437–445 (2013)Google Scholar
  17. 17.
    Wang, S., Chen, Z., Liu, B.: Mining aspect-specific opinion using a holistic lifelong topic model. In: Proceedings of the 25th International Conference on World Wide Web, WWW 2016, pp. 167–176 (2016)Google Scholar
  18. 18.
    Xu, Y., Yin, J., Huang, J., Yin, Y.: Hierarchical topic modeling with automatic knowledge mining. Expert Syst. Appl. 103, 106–117 (2018)CrossRefGoogle Scholar
  19. 19.
    Yan, X., Guo, J., Lan, Y., Cheng, X.: A biterm topic model for short texts. In: Proceedings of the 22nd International Conference on World Wide Web, WWW 2013 (2013)Google Scholar
  20. 20.
    Zhao, P., Li, X., Wang, K.: Feature extraction from micro-blogs for comparison of products and services. In: Lin, X., Manolopoulos, Y., Srivastava, D., Huang, G. (eds.) WISE 2013. LNCS, vol. 8180, pp. 82–91. Springer, Heidelberg (2013). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.The University of QueenslandBrisbaneAustralia
  2. 2.Taibah UniversityMadinahSaudi Arabia
  3. 3.Minia UniversityMiniaEgypt

Personalised recommendations