On How to Perform a Gold Standard Based Evaluation of Ontology Learning

  • Klaas Dellschaft
  • Steffen Staab
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4273)


In recent years several measures for the gold standard based evaluation of ontology learning were proposed. They can be distinguished by the layers of an ontology (e.g. lexical term layer and concept hierarchy) they evaluate. Judging those measures with a list of criteria we show that there exist some measures sufficient for evaluating the lexical term layer. However, existing measures for the evaluation of concept hierarchies fail to meet basic criteria. This paper presents a new taxonomic measure which overcomes the problems of current approaches.


Concept Hierarchy Recall Measure Reference Ontology Ontology Learn Core Ontology 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Cimiano, P., Pivk, A., Schmidt-Thieme, L., Staab, S.: Learning taxonomic relations from heterogenous sources of evidence. In: Ontology Learning from Text: Methods, Applications, Evaluation. IOS Press, Amsterdam (2005)Google Scholar
  2. 2.
    Spyns, P., Reinberger, M.L.: Lexically evaluating ontology triples generated automatically from texts. In: Proc. of the Second European Conference on the Semantic Web (2005)Google Scholar
  3. 3.
    Sabou, M., Wroe, C., Goble, C., Stuckenschmidt, H.: Learning domain ontologies for semantic web service descriptions. Journal of Web Semantics 3(4) (2005)Google Scholar
  4. 4.
    Sabou, M., Wroe, C., Goble, C., Mishne, G.: Learning domain ontologies for web service descriptions: an experiment in bioinformatics. In: Proc. of WWW 2005 (2005)Google Scholar
  5. 5.
    Reinberger, M.L., Spyns, P.: Unsupervised text mining for the learning of dogma-inspired ontologies. In: Ontology Learning from Text: Methods, Applications and EvaluationGoogle Scholar
  6. 6.
    Maedche, A.: Ontology Learning for the Semantic Web. Kluwer, Boston (2002)zbMATHGoogle Scholar
  7. 7.
    Maedche, A., Staab, S.: Measuring similarity between ontologies. In: Gómez-Pérez, A., Benjamins, V.R. (eds.) EKAW 2002. LNCS, vol. 2473. Springer, Heidelberg (2002)Google Scholar
  8. 8.
    Cimiano, P., Hotho, A., Staab, S.: Learning concept hierarchies from text corpora using formal concept analysis. JAIR – Journal of AI Research 24, 305–339 (2005)zbMATHGoogle Scholar
  9. 9.
    Maynard, D., Peters, W., Li, Y.: Metrics for evaluation of ontology-based information extraction. In: Proc. of the EON 2006 Workshop (2006)Google Scholar
  10. 10.
    Hahn, U., Schnattinger, K.: Towards text knowledge engineering. In: Proc. of the 15th National Conference on Artificial Intelligence (AAAI 1998) (1998)Google Scholar
  11. 11.
    Brank, J., Mladenic, D., Grobelnik, M.: Gold standard based ontology evaluation using instance assignment. In: Proc. of the EON 2006 Workshop (2006)Google Scholar
  12. 12.
    Hartmann, J., Spyns, P., Maynard, D., Cuel, R.: Carmen Suarez de Figueroa, M., Sure, Y.: Methods for ontology evaluation. Deliverable D1.2.3, Knowledge Web (2004)Google Scholar
  13. 13.
    van Rijsbergen, C.: Information Retrieval. Butterworths, London (1979)Google Scholar
  14. 14.
    Dellschaft, K.: Measuring the similiarity of concept hierarchies and its influence on the evaluation of learning procedures. Diploma thesis, Universität Koblenz-Landau (2005),
  15. 15.
    Hearst, M.: Automatic aquisition of hyponyms from large text corpora. In: Proc. of the 14th International Conference on Computational Linguistics (1992)Google Scholar
  16. 16.
    Staab, S., Braun, C., Bruder, I., Düsterhöft, A., Heuer, A., Klettke, M., Neumann, G., Prager, B., Pretzel, J., Schnurr, H.-P., Studer, R., Uszkoreit, H., Wrenger, B.: GETESS - searching the web exploiting german texts. In: Klusch, M., Shehory, O., Weiss, G. (eds.) CIA 1999. LNCS, vol. 1652, pp. 113–124. Springer, Heidelberg (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Klaas Dellschaft
    • 1
  • Steffen Staab
    • 1
  1. 1.Universität Koblenz-Landau, ISWeb Working GroupKoblenzGermany

Personalised recommendations