Evaluating Dynamic Ontologies

  • Jaimie Murdock
  • Cameron Buckner
  • Colin Allen
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 272)


Ontology evaluation poses a number of difficult challenges requiring different evaluation methodologies, particularly for a “dynamic ontology” generated by a combination of automatic and semi-automatic methods. We review evaluation methods that focus solely on syntactic (formal) correctness, on the preservation of semantic structure, or on pragmatic utility. We propose two novel methods for dynamic ontology evaluation and describe the use of these methods for evaluating the different taxonomic representations that are generated at different times or with different amounts of expert feedback. These methods are then applied to the Indiana Philosophy Ontology (InPhO), and used to guide the ontology enrichment process.


ontology evaluation taxonomy ontology population dynamic ontology 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Buckner, C., Niepert, M., Allen, C.: From encyclopedia to ontology: Toward dynamic representation of the discipline of philosophy. Synthese (2010)Google Scholar
  2. 2.
    Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: Modelling Ontology Evaluation and Validation. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 140–154. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Guarino, N., Welty, C.A.: An overview of OntoClean. In: Staab, S., Studer, R. (eds.) Handbook on Ontologies, 2nd edn., pp. 151–159. Springer, Heidelberg (2004)Google Scholar
  4. 4.
    Gómez-Pérez, A.: Evaluation of taxonomic knowledge in ontologies and knowledge bases. In: Proceedings of the 12th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Alberta, Canada (1999)Google Scholar
  5. 5.
    Fahad, M., Qadir, M.: A Framework for Ontology Evaluation. In: Proceedings International Conference on Conceptual Structures (ICCS), Toulouse, France, pp. 7–11. Citeseer (July 2008)Google Scholar
  6. 6.
    Dellschaft, K., Staab, S.: Strategies for the Evaluation of Ontology Learning. In: Buitelaar, P., Cimiano, P. (eds.) Ontology Learning and Population: Bridging the Gap Between Text and Knowledge, pp. 253–272. IOS Press (2008)Google Scholar
  7. 7.
    Maedche, A., Staab, S.: Measuring Similarity Between Ontologies. In: Gómez-Pérez, A., Benjamins, V.R. (eds.) EKAW 2002. LNCS (LNAI), vol. 2473, pp. 251–263. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  8. 8.
    Brewster, C., Alani, H., Dasmahapatra, S., Wilks, Y.: Data driven ontology evaluation. In: Proceedings of LREC, vol. 2004 (2004)Google Scholar
  9. 9.
    Supekar, K.: A peer-review approach for ontology evaluation. In: 8th Int. Protege Conf., pp. 77–79. Citeseer (2004)Google Scholar
  10. 10.
    Staab, S., Gómez-Pérez, A., Daelemans, W., Reinberger, M.L., Guarino, N., Noy, N.F.: Why evaluate ontology technologies? because it works! IEEE Intelligent Systems 19, 74–81 (2004)Google Scholar
  11. 11.
    Lozano-Tello, A., Gómez-Pérez, A.: Ontometric: A method to choose the appropriate ontology. Journal of Database Management 15, 1–18 (2004)CrossRefGoogle Scholar
  12. 12.
    Brank, J., Grobelnik, M., Mladenic, D.: Survey of ontology evaluation techniques. In: Proceedings of the Conference on Data Mining and Data Warehouses (SiKDD) (2005)Google Scholar
  13. 13.
    Velardi, P., Navigli, R., Cucchiarelli, A., Neri, F.: Evaluation of OntoLearn, a methodology for automatic learning of domain ontologies. In: Buitelaar, P., Cimiano, P., Magnini, B. (eds.) Ontology Learning from Text: Methods, Evaluation and Applications. IOS Press, Amsterdam (2005)Google Scholar
  14. 14.
    Porzel, R., Malaka, R.: A task-based framework for ontology learning, population and evaluation. In: Buitelaar, P., Cimiano, P., Magnini, B. (eds.) Ontology Learning from Text: Methods, Evaluation and Applications. IOS Press, Amsterdam (2005)Google Scholar
  15. 15.
    Gruber, T.R.: Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human Computer Studies 43, 907–928 (1995)CrossRefGoogle Scholar
  16. 16.
    Noy, N., McGuinness, D.: Ontology development 101: A guide to creating your first ontology (2001)Google Scholar
  17. 17.
    Niepert, M., Buckner, C., Allen, C.: A dynamic ontology for a dynamic reference work. In: Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, p. 297. ACM (2007)Google Scholar
  18. 18.
    Niepert, M., Buckner, C., Allen, C.: Answer set programming on expert feedback to populate and extend dynamic ontologies. In: Proceedings of 21st FLAIRS (2008)Google Scholar
  19. 19.
    Smyth, P., Goodman, R.M.: An information theoretic approach to rule induction from databases. IEEE Transactions on Knowledge and Data Engineering 4, 301–316 (1992)CrossRefGoogle Scholar
  20. 20.
    Shannon, C.E.: A mathematical theory of communication. University of Illinois Press, Urbana (1949)Google Scholar
  21. 21.
    Smith, B.: Ontology. In: Luciano, F. (ed.) Blackweel Guide to the Philosophy of Computing and Information, pp. 155–166. Blackwell, Oxford (2003)Google Scholar
  22. 22.
    Kuhn, T.: The Structure of Scientific Revolutions. University of Chicago Press (1962)Google Scholar
  23. 23.
    Jiang, J., Conrath, D.: Semantic similarity based on corpus statistics and lexical taxonomy. In: Proceedings of International Conference Research on Computational Linguistics (ROCLING X), Number Rocling X, Taiwan (1997)Google Scholar
  24. 24.
    Resnik, P.: Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research 11, 95–130 (1999)zbMATHGoogle Scholar
  25. 25.
    Lin, D.: An information-theoretic definition of similarity. In: Proceedings of the 15th International Conference on Machine Learning, pp. 296–304. Citeseer (1998)Google Scholar
  26. 26.
    Eckert, K., Niepert, M., Niemann, C., Buckner, C., Allen, C., Stuckenschmidt, H.: Crowdsourcing the Assembly of Concept Hierarchies. In: Proceedings of the 10th ACM/IEEE Joint Conference on Digital Libraries (JCDL), Brisbane, Australia. ACM Press (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Jaimie Murdock
    • 1
  • Cameron Buckner
    • 1
  • Colin Allen
    • 1
  1. 1.Indiana UniversityBloomingtonU.S A.

Personalised recommendations