Ontology Evaluation

Chapter
First Online:
Part of the International Handbooks on Information Systems book series (INFOSYS)

Summary

The evaluation of ontologies is still an emerging field. A set of preliminary ideas and frameworks have been suggested in the literature. This chapter collects ontology quality criteria and lays out a common framework for aspects of ontology evaluation. It will present in depth descriptions of these ontology aspects and how to evaluate them. The techniques and ideas collected and presented here will help to uncover errors in ontologies. This chapter concentrates on the automatic, domain- and task-independent evaluation of an ontology.

Keywords

Ontology Language Ontology Engineering Ontology Learning Competency Question Ontology Graph 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access

Notes

Acknowledgements

The ideas presented in this chapter have been collected and explored in discussions with many others. I want to especially thank York Sure, Aldo Gangemi, Philipp Cimiano, Johanna Völker, Enrico Motta, Elena Paslaru Bontas Simperl, Marta Sabou, Markus Krötzsch, Anupriya Ankolekar, Boris Motik, Stephan Grimm, Hans-Jörg Happel, Malvina Nissim, Jos Lehmann, Valentina Presutti, Joey Lam, Christoph Tempich, Peter Haase, Dan Connolly, Rinke Hoekstra, Sebastian Rudolph, Chris Welty, Aleks Jakulin, and the participants of the EON 2006 workshop in Edinburgh, Scotland, and the EON 2007 workshop in Busan, South Korea.

References

  1. 1.
    Harith Alani. Position paper: Ontology construction from online ontologies. In Les Carr, David De Roure, Arun Iyengar, Carole A. Goble, and Michael Dahlin, editors, Proceedings of the 15th International Conference on World Wide Web, WWW 2006, Edinburgh, Scotland, UK, May 23–26, 2006, pages 491–495. ACM, New York, 2006.Google Scholar
  2. 2.
    Harith Alani and Christopher Brewster. Metrics for ranking ontologies. In Denny Vrandečić, Mari del Carmen Suárez-Figueroa, Aldo Gangemi, and York Sure, editors. Proceedings of the 4th International Workshop on Evaluation of Ontologies for the Web (EON2006) at the 15th International World Wide Web Conference (WWW 2006), Edinburgh, Scotland, May 2006, pages 24–30.Google Scholar
  3. 3.
    Sean Bechhofer, Frank van Harmelen, Jim Hendler, Ian Horrocks, Deborah L. McGuinness, Peter F. Patel-Schneider, and Lynn Andrea Stein. OWL Web Ontology Language Abstract Reference, 2004. W3C Rec. 10 February 2004.Google Scholar
  4. 4.
    Dave Beckett. RDF/XML syntax specification (revised). W3C Recommendation, February 2004.Google Scholar
  5. 5.
    Tim Berners-Lee. Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web. Technical Report 1630, Internet Engineering Task Force, June 1994.Google Scholar
  6. 6.
    Tim Berners-Lee, James Hendler, and Ora Lassila. The Semantic Web. Scientific American, 284(5):34–43, 2001.CrossRefGoogle Scholar
  7. 7.
    Tim Bray, Dave Hollander, Andrew Layman, and Richard Tobin. Namespaces in XML 1.0 (second edition), 2006. W3C Rec. 16 August 2006.Google Scholar
  8. 8.
    Tim Bray, Jean Paoli, C. Michael Sperberg-McQueen, Eve Maler, and Francois Yergeau. Extensible markup language (XML) 1.0 (fourth edition), 2006. W3C Rec. 16 August 2006.Google Scholar
  9. 9.
    Christopher Brewster, Harith Alani, Srinandan Dasmahapatra, and Yorick Wilks. Data-driven ontology evaluation. In Proceedings of the Language Resources and Evaluation Conference (LREC 2004), pages 164–168, Lisbon, Portugal, 2004. European Language Resources Association.Google Scholar
  10. 10.
    Dan Brickley and Libby Miller. The Friend Of A Friend (FOAF) vocabulary specification, July 2005.Google Scholar
  11. 11.
    James Clark and Makoto Murata. RELAX NG Specification, December 2001. OASIS committee specification.Google Scholar
  12. 12.
    Mathieau d’Aquin, Claudio Baldassarre, Larian Gridinoc, Sofia Angeletou, Marta Sabou, and Enrico Motta. Characterizing knowledge on the semantic web with watson. In Denny Vrandečić, Raúl García-Castro, Asunción Gómez-Pérez, York Sure, and Zhisheng Huang, editors, Proceedings of the Workshop on Evaluation of Ontologies and Ontology-Based Tools, 5th International EON Workshop (EON2007) at ISWC/ASWC’07, pages 1–10, Busan, Korea, November 2007.Google Scholar
  13. 13.
    Li Ding, Tim Finin, Anupam Joshi, Rong Pan, R. Scott Cost, Yun Peng, Pavan Reddivari, Vishal Doshi, and Joel Sachs. Swoogle: A search and metadata engine for the semantic web. In CIKM’04: Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, pages 652–659. ACM, New York, NY, 2004.Google Scholar
  14. 14.
    David C. Fallside and Priscilla Walmsley. XML schema part 0: Primer second edition, 2004. W3C Rec. 28 October 2004.Google Scholar
  15. 15.
    Christine Fellbaum. WordNet: An Electronic Lexical Database (Language, Speech, and Communication). MIT, New York, 1998.MATHGoogle Scholar
  16. 16.
    Roy Fielding, James Gettys, Jeffrey Mogul, Henrik Frystyk, Larry Masinter, Paul Leach, and Tim Berners-Lee. Hypertext Transfer Protocol – HTTP/1.1. RFC 2616, June 1999.Google Scholar
  17. 17.
    Aldo Gangemi, Carola Catenacci, Massimiliano Ciaramita, and Jens Lehmann. Ontology evaluation and validation: An integrated formal model for the quality diagnostic task. Technical report, Laboratory of Applied Ontologies – CNR, Rome, Italy, 2005. http://www.loa-cnr.it/Files/OntoEval4OntoDev_Final.pdf.
  18. 18.
    Aldo Gangemi, Carola Catenaccia, Massimiliano Ciaramita, and Jos Lehmann. Qood grid: A metaontology-based framework for ontology evaluation and selection. In Denny Vrandečić, Mari del Carmen Suárez-Figueroa, Aldo Gangemi, and York Sure, editors. Proceedings of the 4th International Workshop on Evaluation of Ontologies for the Web (EON2006) at the 15th International World Wide Web Conference (WWW 2006), Edinburgh, Scotland, May 2006, pages 8–15.Google Scholar
  19. 19.
    Asunción Gómez-Pérez. Ontology evaluation. In Steffen Staab and Rudi Studer, editors, Handbook on Ontologies in Information Systems, First Edition, International Handbooks on Information Systems, chapter 13, pages 251–274. Springer, Berlin, 2004.Google Scholar
  20. 20.
    Jan Grant and Dave Beckett. RDF test cases. W3C Recommendation, February 2004.Google Scholar
  21. 21.
    Stephan Grimm and Boris Motik. Closed world reasoning in the semantic web through epistemic operators. In Bernardo Cuenca Grau, Ian Horrocks, Bijan Parsia, and Peter Patel-Schneider, editors, Second International Workshop on OWL: Experiences and Directions (OWLED 2006), Galway, Ireland, 2005.Google Scholar
  22. 22.
    Thomas R. Gruber. Towards principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies, 43(5/6):907–928, 1995.CrossRefGoogle Scholar
  23. 23.
    Michael Grüninger and Mark S. Fox. Methodology for the design and evaluation of ontologies. In IJCAI95 Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal, 1995.Google Scholar
  24. 24.
    Peter Haase and Guilin Qi. An analysis of approaches to resolving inconsistencies in DL-based ontologies. In Proceedings of International Workshop on Ontology Dynamics (IWOD’07), pages 97–109, June 2007.Google Scholar
  25. 25.
    Masahiro Hori, Jérôme Euzenat, and Peter F. Patel-Schneider. OWL Web Ontology Language XML presentation syntax, 2003. W3C Note 11 June 2003.Google Scholar
  26. 26.
    Matthew Horridge. The Protégé OWL unit test framework, 2005. Website at http://www.co-ode.org/downloads/owlunittest/.
  27. 27.
    Matthew Horridge, Nick Drummond, John Goodwin, Alan Rector, Robert Stevens, and Hai Wang. The manchester owl syntax. In OWLED2006 Second Workshop on OWL Experiences and Directions, Athens, GA, USA, 2006.Google Scholar
  28. 28.
    Ian Horrocks and Peter F. Patel-Schneider. Reducing OWL Entailment to Description Logic Satisfiability. Journal of Web Semantics, 1(4):7–26, 2004.CrossRefGoogle Scholar
  29. 29.
    Aleks Jakulin and Dunja Mladenić. Ontology grounding. In Proceedings of 8th International Multi-conference Information Society IS-2005, pages 170–173, 2005.Google Scholar
  30. 30.
    Aditya Kalyanpur, Bijan Parsia, Evren Sirin, Bernardo Cuenca Grau, and James Hendler. Swoop: A web ontology editing browser. Journal of Web Semantics: Science, Services and Agents on the World Wide Web, 4(2):144–153, June 2006.Google Scholar
  31. 31.
    Graham Klyne and Jeremy Carroll. Resource Description Framework (RDF): Concepts and abstract syntax. W3C Recommendation 10 February 2004.Google Scholar
  32. 32.
    Konstantinos Kotis, George A. Vouros, and Jerónimo P. Alonso. HCOME: A tool-supported methodology for engineering living ontologies. Semantic Web and Databases, pages 155–166, 2005.Google Scholar
  33. 33.
    Markus Krötzsch, Denny Vrandečić, Max Völkel, Heiko Haller, and Rudi Studer. Semantic wikipedia. Journal of Web Semantics, 5:251–261, 2007.CrossRefGoogle Scholar
  34. 34.
    Joey Lam. Methods for Resolving Inconsistencies in Ontologies. PhD thesis, University of Aberdeen, 2007.Google Scholar
  35. 35.
    Adolfo Lozano-Tello and Asunción Gómez-Pérez. OntoMetric: A method to choose the appropriate ontology. Journal of Database Management Special Issue on Ontological analysis, Evaluation, and Engineering of Business Systems Analysis Methods, 15(2), 2004.Google Scholar
  36. 36.
    Boris Motik and Ian Horrocks. Problems with OWL syntax. In OWLED2006 Second Workshop on OWL Experiences and Directions, Athens, GA, 2006.Google Scholar
  37. 37.
    Boris Motik, Ian Horrocks, and Ulrike Sattler. Adding Integrity Constraints to OWL. In Christine Golbreich, Aditya Kalyanpur, and Bijan Parsia, editors, Third International Workshop on OWL: Experiences and Directions 2007 (OWLED 2007), Innsbruck, Austria, June 6–7, 2007.Google Scholar
  38. 38.
    Leo Obrst, Werner Ceusters, Inderjeet Mani, Steve Ray, and Barry Smith. The evaluation of ontologies. In Christopher J.O. Baker and Kei-Hoi Cheung, editors, Revolutionizing Knowledge Discovery in the Life Sciences, chapter 7, pages 139–158. Springer, Berlin, 2007.Google Scholar
  39. 39.
    Eyal Oren, Renaud Delbru, Michele Catasta, Richard Cyganiak, Holger Stenzhorn, and Giovannia Tummarello. Sindice.com: A document-oriented lookup index for open linked data. International Journal of Metadata, Semantics and Ontologies, 3(1), 2008.Google Scholar
  40. 40.
    Chintan Patel, Kaustubh Supekar, Yugyung Lee, and E. K. Park. OntoKhoj: A semantic web portal for ontology searching, ranking and classification. In Proceedings of Fifth ACM International Workshop on Web Information and Data Management, pages 58–61, New York, NY, 2003.Google Scholar
  41. 41.
    Peter Patel-Schneider, Patrick Hayes, and Ian Horrocks. OWL Web Ontology Language Abstract Syntax and Semantics, 2004. W3C Rec. 10 February 2004.Google Scholar
  42. 42.
    Addison Phillips and Mark Davis. Tags for Identifying Languages. Technical Report 4646, Internet Engineering Task Force, September 2006.Google Scholar
  43. 43.
    Robert Porzel and Rainer Malaka. A task-based approach for ontology evaluation. In Paul Buitelaar, Siegrfried Handschuh, and Bernardo Magnini, editors, Proceedings of ECAI 2004 Workshop on Ontology Learning and Population, Valencia, Spain, August 2004.Google Scholar
  44. 44.
    Sebastian Rudolph, Johanna Völker, and Pascal Hitzler. Supporting lexical ontology learning by relational exploration. In Uta Priss, Simon Polovina, and Richard Hill, editors, Conceptual Structures: Knowledge Architectures for Smart Applications, Proc. ICCS 2007, volume 4604 of LNAI, pages 488–491. Springer, Sheffield, UK, 2007.Google Scholar
  45. 45.
    Marta Sabou, Jorge Gracia, Sofia Angeletou, Mathieu d’Aquin, and Enrico Motta. Evaluating the semantic web: A task-based approach. In Proceedings of the 6th International Semantic Web Conference (ISWC’07), pages 423–437, November 2007.Google Scholar
  46. 46.
    Leo Sauermann and Richard Cyganiak. Cool URIs for the semantic web. Interest group note, W3C, March 2008.Google Scholar
  47. 47.
    Andrea Schaerf. Reasoning with individuals in concept languages. Data and Knowlegde Engineering, 13(2):141–176, 1994.CrossRefGoogle Scholar
  48. 48.
    Michael K. Smith, Chris Welty, and Deborah McGuinness. OWL Web Ontology Language Guide, 2004. W3C Rec. 10 February 2004.Google Scholar
  49. 49.
    Aaron Swartz. RDF site summary (RSS) 1.0, December 2000. Official specification.Google Scholar
  50. 50.
    Samir Tartir, I. Budak Arpinar, Michael Moore, Amit P. Sheth, and Boanerges Aleman-Meza. OntoQA: Metric-based ontology quality analysis. In Proceedings of IEEE Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources, pages 45–53, 2005.Google Scholar
  51. 51.
    Christoph Tempich, Helena Sofia Pinto, York Sure, and Steffen Staab. An argumentation ontology for DIstributed, Loosely-controlled and evolvInG Engineering processes of oNTologies (DILIGENT). In Asunción Gómez-Pérez and Jérôme Euzenat, editors, Proceedings of the Second European Semantic Web Conference, volume 3532, pages 241–256, Heraklion, Crete, Greece. Springer, Berlin, 2005.Google Scholar
  52. 52.
    Paola Velardi, Roberto Navigli, Alessandro Cucchiarelli, and Francesca Neri. Evaluation of OntoLearn, a methodology for automatic population of domain ontologies. In Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini, editors, Ontology Learning from Text: Methods, Applications and Evaluation. IOS Press, Amsterdam, 2005.Google Scholar
  53. 53.
    Johanna Völker, Denny Vrandečić, and York Sure. Automatic evaluation of ontologies (AEON). In Yolanda Gil, Enrico Motta, V. Richard Benjamins, and Mark A. Musen, editors, Proceedings of the Fourth International Semantic Web Conference (ISWC’05), volume 3729 of LNCS. Springer, Berlin, 2005.Google Scholar
  54. 54.
    Denny Vrandečić. Explicit knowledge engineering patterns with macros. In Chris Welty and Aldo Gangemi, editors, Proceedings of the Ontology Patterns for the Semantic Web Workshop at the ISWC 2005, Galway, Ireland, November 2005.Google Scholar
  55. 55.
    Denny Vrandečić and Aldo Gangemi. Unit tests for ontologies. In Mustafa Jarrar, Claude Ostyn, Werner Ceusters, and Andreas Persidis, editors, Proceedings of the 1th International Workshop on Ontology Content and Evaluation in Enterprise, LNCS, Montpellier, France, October. Springer, Berlin, 2006.Google Scholar
  56. 56.
    Denny Vrandečić and Aldo Gangemi. Unit tests for ontologies. In Mustafa Jarrar, Claude Ostyn, Werner Ceusters, and Andreas Persidis, editors, Proceedings of the 1st International Workshop on Ontology Content and Evaluation in Enterprise, LNCS, Montpellier, France, October. Springer, Berlin, 2006.Google Scholar
  57. 57.
    Denny Vrandečić and York Sure. How to design better ontology metrics. In Wolfgang May and Michael Kifer, editors, Proceedings of the 4th European Semantic Web Conference (ESWC’07), Innsbruck, Austria. Springer, Berlin, 2007.Google Scholar
  58. 58.
    Denny Vrandečić, Johanna Völker, Peter Haase, Duc Thanh Tran, and Philipp Cimiano. A metamodel for annotations of ontology elements in OWL DL. In York Sure, Saartje Brockmans, and Jürgen Jung, editors, Proceedings of the Second Workshop on Ontologies and Meta-Modeling, Karlsruhe, Germany, October 2006. GI Gesellschaft für Informatik.Google Scholar
  59. 59.
    Taowei David Wang and Bijan Parsia. Ontology performance profiling and model examination: First steps. In Karl Aberer, Key-Sun Choi, Natasha Fridman Noy, Dean Allemang, Kyung-Il Lee, Lyndon J. B. Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Mauroux, editors, Proceedings of the 6th International Semantic Web Conference/2nd Asian Semantic Web Conference (ISWC/ASWC’07), volume 4825 of Lecture Notes in Computer Science, pages 595–608, Busan, Korea, November. Springer, Berlin, 2007.Google Scholar
  60. 60.
    Taowei David Wang, Bijan Parsia, and James A. Hendler. A survey of the web ontology landscape. In Isabel Cruz, Stefan Decker, Dean Allemang Chris Preist, Daniel Schwabe, Peter Mika, Michael Uschold, and Lora Aroyo, editors, Proceedings of the Fifth International Semantic Web Conference (ISWC’06), volume 4273 of Lecture Notes in Computer Science, pages 682–694, Athens, Georgia, November 2006. Springer.Google Scholar
  61. 61.
    Jonathan Yu, James A. Thom, and Audrey Tam. Ontology evaluation using wikipedia categories for browsing. In Proceedings of the ACM Sixteenth Conference on Information and Knowledge Management (CIKM), Lisboa, Portugal, November 2007. ACM.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  1. 1.Institut AIFBUniversität Karlsruhe (TH)KarlsruheGermany

Personalised recommendations