Handbook on Ontologies pp 293-313 | Cite as
Ontology Evaluation
Summary
The evaluation of ontologies is still an emerging field. A set of preliminary ideas and frameworks have been suggested in the literature. This chapter collects ontology quality criteria and lays out a common framework for aspects of ontology evaluation. It will present in depth descriptions of these ontology aspects and how to evaluate them. The techniques and ideas collected and presented here will help to uncover errors in ontologies. This chapter concentrates on the automatic, domain- and task-independent evaluation of an ontology.
Keywords
Ontology Language Ontology Engineering Ontology Learning Competency Question Ontology GraphNotes
Acknowledgements
The ideas presented in this chapter have been collected and explored in discussions with many others. I want to especially thank York Sure, Aldo Gangemi, Philipp Cimiano, Johanna Völker, Enrico Motta, Elena Paslaru Bontas Simperl, Marta Sabou, Markus Krötzsch, Anupriya Ankolekar, Boris Motik, Stephan Grimm, Hans-Jörg Happel, Malvina Nissim, Jos Lehmann, Valentina Presutti, Joey Lam, Christoph Tempich, Peter Haase, Dan Connolly, Rinke Hoekstra, Sebastian Rudolph, Chris Welty, Aleks Jakulin, and the participants of the EON 2006 workshop in Edinburgh, Scotland, and the EON 2007 workshop in Busan, South Korea.
References
- 1.Harith Alani. Position paper: Ontology construction from online ontologies. In Les Carr, David De Roure, Arun Iyengar, Carole A. Goble, and Michael Dahlin, editors, Proceedings of the 15th International Conference on World Wide Web, WWW 2006, Edinburgh, Scotland, UK, May 23–26, 2006, pages 491–495. ACM, New York, 2006.Google Scholar
- 2.Harith Alani and Christopher Brewster. Metrics for ranking ontologies. In Denny Vrandečić, Mari del Carmen Suárez-Figueroa, Aldo Gangemi, and York Sure, editors. Proceedings of the 4th International Workshop on Evaluation of Ontologies for the Web (EON2006) at the 15th International World Wide Web Conference (WWW 2006), Edinburgh, Scotland, May 2006, pages 24–30.Google Scholar
- 3.Sean Bechhofer, Frank van Harmelen, Jim Hendler, Ian Horrocks, Deborah L. McGuinness, Peter F. Patel-Schneider, and Lynn Andrea Stein. OWL Web Ontology Language Abstract Reference, 2004. W3C Rec. 10 February 2004.Google Scholar
- 4.Dave Beckett. RDF/XML syntax specification (revised). W3C Recommendation, February 2004.Google Scholar
- 5.Tim Berners-Lee. Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web. Technical Report 1630, Internet Engineering Task Force, June 1994.Google Scholar
- 6.Tim Berners-Lee, James Hendler, and Ora Lassila. The Semantic Web. Scientific American, 284(5):34–43, 2001.CrossRefGoogle Scholar
- 7.Tim Bray, Dave Hollander, Andrew Layman, and Richard Tobin. Namespaces in XML 1.0 (second edition), 2006. W3C Rec. 16 August 2006.Google Scholar
- 8.Tim Bray, Jean Paoli, C. Michael Sperberg-McQueen, Eve Maler, and Francois Yergeau. Extensible markup language (XML) 1.0 (fourth edition), 2006. W3C Rec. 16 August 2006.Google Scholar
- 9.Christopher Brewster, Harith Alani, Srinandan Dasmahapatra, and Yorick Wilks. Data-driven ontology evaluation. In Proceedings of the Language Resources and Evaluation Conference (LREC 2004), pages 164–168, Lisbon, Portugal, 2004. European Language Resources Association.Google Scholar
- 10.Dan Brickley and Libby Miller. The Friend Of A Friend (FOAF) vocabulary specification, July 2005.Google Scholar
- 11.James Clark and Makoto Murata. RELAX NG Specification, December 2001. OASIS committee specification.Google Scholar
- 12.Mathieau d’Aquin, Claudio Baldassarre, Larian Gridinoc, Sofia Angeletou, Marta Sabou, and Enrico Motta. Characterizing knowledge on the semantic web with watson. In Denny Vrandečić, Raúl García-Castro, Asunción Gómez-Pérez, York Sure, and Zhisheng Huang, editors, Proceedings of the Workshop on Evaluation of Ontologies and Ontology-Based Tools, 5th International EON Workshop (EON2007) at ISWC/ASWC’07, pages 1–10, Busan, Korea, November 2007.Google Scholar
- 13.Li Ding, Tim Finin, Anupam Joshi, Rong Pan, R. Scott Cost, Yun Peng, Pavan Reddivari, Vishal Doshi, and Joel Sachs. Swoogle: A search and metadata engine for the semantic web. In CIKM’04: Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, pages 652–659. ACM, New York, NY, 2004.Google Scholar
- 14.David C. Fallside and Priscilla Walmsley. XML schema part 0: Primer second edition, 2004. W3C Rec. 28 October 2004.Google Scholar
- 15.Christine Fellbaum. WordNet: An Electronic Lexical Database (Language, Speech, and Communication). MIT, New York, 1998.zbMATHGoogle Scholar
- 16.Roy Fielding, James Gettys, Jeffrey Mogul, Henrik Frystyk, Larry Masinter, Paul Leach, and Tim Berners-Lee. Hypertext Transfer Protocol – HTTP/1.1. RFC 2616, June 1999.Google Scholar
- 17.Aldo Gangemi, Carola Catenacci, Massimiliano Ciaramita, and Jens Lehmann. Ontology evaluation and validation: An integrated formal model for the quality diagnostic task. Technical report, Laboratory of Applied Ontologies – CNR, Rome, Italy, 2005. http://www.loa-cnr.it/Files/OntoEval4OntoDev_Final.pdf.
- 18.Aldo Gangemi, Carola Catenaccia, Massimiliano Ciaramita, and Jos Lehmann. Qood grid: A metaontology-based framework for ontology evaluation and selection. In Denny Vrandečić, Mari del Carmen Suárez-Figueroa, Aldo Gangemi, and York Sure, editors. Proceedings of the 4th International Workshop on Evaluation of Ontologies for the Web (EON2006) at the 15th International World Wide Web Conference (WWW 2006), Edinburgh, Scotland, May 2006, pages 8–15.Google Scholar
- 19.Asunción Gómez-Pérez. Ontology evaluation. In Steffen Staab and Rudi Studer, editors, Handbook on Ontologies in Information Systems, First Edition, International Handbooks on Information Systems, chapter 13, pages 251–274. Springer, Berlin, 2004.Google Scholar
- 20.Jan Grant and Dave Beckett. RDF test cases. W3C Recommendation, February 2004.Google Scholar
- 21.Stephan Grimm and Boris Motik. Closed world reasoning in the semantic web through epistemic operators. In Bernardo Cuenca Grau, Ian Horrocks, Bijan Parsia, and Peter Patel-Schneider, editors, Second International Workshop on OWL: Experiences and Directions (OWLED 2006), Galway, Ireland, 2005.Google Scholar
- 22.Thomas R. Gruber. Towards principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies, 43(5/6):907–928, 1995.CrossRefGoogle Scholar
- 23.Michael Grüninger and Mark S. Fox. Methodology for the design and evaluation of ontologies. In IJCAI95 Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal, 1995.Google Scholar
- 24.Peter Haase and Guilin Qi. An analysis of approaches to resolving inconsistencies in DL-based ontologies. In Proceedings of International Workshop on Ontology Dynamics (IWOD’07), pages 97–109, June 2007.Google Scholar
- 25.Masahiro Hori, Jérôme Euzenat, and Peter F. Patel-Schneider. OWL Web Ontology Language XML presentation syntax, 2003. W3C Note 11 June 2003.Google Scholar
- 26.Matthew Horridge. The Protégé OWL unit test framework, 2005. Website at http://www.co-ode.org/downloads/owlunittest/.
- 27.Matthew Horridge, Nick Drummond, John Goodwin, Alan Rector, Robert Stevens, and Hai Wang. The manchester owl syntax. In OWLED2006 Second Workshop on OWL Experiences and Directions, Athens, GA, USA, 2006.Google Scholar
- 28.Ian Horrocks and Peter F. Patel-Schneider. Reducing OWL Entailment to Description Logic Satisfiability. Journal of Web Semantics, 1(4):7–26, 2004.CrossRefGoogle Scholar
- 29.Aleks Jakulin and Dunja Mladenić. Ontology grounding. In Proceedings of 8th International Multi-conference Information Society IS-2005, pages 170–173, 2005.Google Scholar
- 30.Aditya Kalyanpur, Bijan Parsia, Evren Sirin, Bernardo Cuenca Grau, and James Hendler. Swoop: A web ontology editing browser. Journal of Web Semantics: Science, Services and Agents on the World Wide Web, 4(2):144–153, June 2006.Google Scholar
- 31.Graham Klyne and Jeremy Carroll. Resource Description Framework (RDF): Concepts and abstract syntax. W3C Recommendation 10 February 2004.Google Scholar
- 32.Konstantinos Kotis, George A. Vouros, and Jerónimo P. Alonso. HCOME: A tool-supported methodology for engineering living ontologies. Semantic Web and Databases, pages 155–166, 2005.Google Scholar
- 33.Markus Krötzsch, Denny Vrandečić, Max Völkel, Heiko Haller, and Rudi Studer. Semantic wikipedia. Journal of Web Semantics, 5:251–261, 2007.CrossRefGoogle Scholar
- 34.Joey Lam. Methods for Resolving Inconsistencies in Ontologies. PhD thesis, University of Aberdeen, 2007.Google Scholar
- 35.Adolfo Lozano-Tello and Asunción Gómez-Pérez. OntoMetric: A method to choose the appropriate ontology. Journal of Database Management Special Issue on Ontological analysis, Evaluation, and Engineering of Business Systems Analysis Methods, 15(2), 2004.Google Scholar
- 36.Boris Motik and Ian Horrocks. Problems with OWL syntax. In OWLED2006 Second Workshop on OWL Experiences and Directions, Athens, GA, 2006.Google Scholar
- 37.Boris Motik, Ian Horrocks, and Ulrike Sattler. Adding Integrity Constraints to OWL. In Christine Golbreich, Aditya Kalyanpur, and Bijan Parsia, editors, Third International Workshop on OWL: Experiences and Directions 2007 (OWLED 2007), Innsbruck, Austria, June 6–7, 2007.Google Scholar
- 38.Leo Obrst, Werner Ceusters, Inderjeet Mani, Steve Ray, and Barry Smith. The evaluation of ontologies. In Christopher J.O. Baker and Kei-Hoi Cheung, editors, Revolutionizing Knowledge Discovery in the Life Sciences, chapter 7, pages 139–158. Springer, Berlin, 2007.Google Scholar
- 39.Eyal Oren, Renaud Delbru, Michele Catasta, Richard Cyganiak, Holger Stenzhorn, and Giovannia Tummarello. Sindice.com: A document-oriented lookup index for open linked data. International Journal of Metadata, Semantics and Ontologies, 3(1), 2008.Google Scholar
- 40.Chintan Patel, Kaustubh Supekar, Yugyung Lee, and E. K. Park. OntoKhoj: A semantic web portal for ontology searching, ranking and classification. In Proceedings of Fifth ACM International Workshop on Web Information and Data Management, pages 58–61, New York, NY, 2003.Google Scholar
- 41.Peter Patel-Schneider, Patrick Hayes, and Ian Horrocks. OWL Web Ontology Language Abstract Syntax and Semantics, 2004. W3C Rec. 10 February 2004.Google Scholar
- 42.Addison Phillips and Mark Davis. Tags for Identifying Languages. Technical Report 4646, Internet Engineering Task Force, September 2006.Google Scholar
- 43.Robert Porzel and Rainer Malaka. A task-based approach for ontology evaluation. In Paul Buitelaar, Siegrfried Handschuh, and Bernardo Magnini, editors, Proceedings of ECAI 2004 Workshop on Ontology Learning and Population, Valencia, Spain, August 2004.Google Scholar
- 44.Sebastian Rudolph, Johanna Völker, and Pascal Hitzler. Supporting lexical ontology learning by relational exploration. In Uta Priss, Simon Polovina, and Richard Hill, editors, Conceptual Structures: Knowledge Architectures for Smart Applications, Proc. ICCS 2007, volume 4604 of LNAI, pages 488–491. Springer, Sheffield, UK, 2007.Google Scholar
- 45.Marta Sabou, Jorge Gracia, Sofia Angeletou, Mathieu d’Aquin, and Enrico Motta. Evaluating the semantic web: A task-based approach. In Proceedings of the 6th International Semantic Web Conference (ISWC’07), pages 423–437, November 2007.Google Scholar
- 46.Leo Sauermann and Richard Cyganiak. Cool URIs for the semantic web. Interest group note, W3C, March 2008.Google Scholar
- 47.Andrea Schaerf. Reasoning with individuals in concept languages. Data and Knowlegde Engineering, 13(2):141–176, 1994.CrossRefGoogle Scholar
- 48.Michael K. Smith, Chris Welty, and Deborah McGuinness. OWL Web Ontology Language Guide, 2004. W3C Rec. 10 February 2004.Google Scholar
- 49.Aaron Swartz. RDF site summary (RSS) 1.0, December 2000. Official specification.Google Scholar
- 50.Samir Tartir, I. Budak Arpinar, Michael Moore, Amit P. Sheth, and Boanerges Aleman-Meza. OntoQA: Metric-based ontology quality analysis. In Proceedings of IEEE Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources, pages 45–53, 2005.Google Scholar
- 51.Christoph Tempich, Helena Sofia Pinto, York Sure, and Steffen Staab. An argumentation ontology for DIstributed, Loosely-controlled and evolvInG Engineering processes of oNTologies (DILIGENT). In Asunción Gómez-Pérez and Jérôme Euzenat, editors, Proceedings of the Second European Semantic Web Conference, volume 3532, pages 241–256, Heraklion, Crete, Greece. Springer, Berlin, 2005.Google Scholar
- 52.Paola Velardi, Roberto Navigli, Alessandro Cucchiarelli, and Francesca Neri. Evaluation of OntoLearn, a methodology for automatic population of domain ontologies. In Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini, editors, Ontology Learning from Text: Methods, Applications and Evaluation. IOS Press, Amsterdam, 2005.Google Scholar
- 53.Johanna Völker, Denny Vrandečić, and York Sure. Automatic evaluation of ontologies (AEON). In Yolanda Gil, Enrico Motta, V. Richard Benjamins, and Mark A. Musen, editors, Proceedings of the Fourth International Semantic Web Conference (ISWC’05), volume 3729 of LNCS. Springer, Berlin, 2005.Google Scholar
- 54.Denny Vrandečić. Explicit knowledge engineering patterns with macros. In Chris Welty and Aldo Gangemi, editors, Proceedings of the Ontology Patterns for the Semantic Web Workshop at the ISWC 2005, Galway, Ireland, November 2005.Google Scholar
- 55.Denny Vrandečić and Aldo Gangemi. Unit tests for ontologies. In Mustafa Jarrar, Claude Ostyn, Werner Ceusters, and Andreas Persidis, editors, Proceedings of the 1th International Workshop on Ontology Content and Evaluation in Enterprise, LNCS, Montpellier, France, October. Springer, Berlin, 2006.Google Scholar
- 56.Denny Vrandečić and Aldo Gangemi. Unit tests for ontologies. In Mustafa Jarrar, Claude Ostyn, Werner Ceusters, and Andreas Persidis, editors, Proceedings of the 1st International Workshop on Ontology Content and Evaluation in Enterprise, LNCS, Montpellier, France, October. Springer, Berlin, 2006.Google Scholar
- 57.Denny Vrandečić and York Sure. How to design better ontology metrics. In Wolfgang May and Michael Kifer, editors, Proceedings of the 4th European Semantic Web Conference (ESWC’07), Innsbruck, Austria. Springer, Berlin, 2007.Google Scholar
- 58.Denny Vrandečić, Johanna Völker, Peter Haase, Duc Thanh Tran, and Philipp Cimiano. A metamodel for annotations of ontology elements in OWL DL. In York Sure, Saartje Brockmans, and Jürgen Jung, editors, Proceedings of the Second Workshop on Ontologies and Meta-Modeling, Karlsruhe, Germany, October 2006. GI Gesellschaft für Informatik.Google Scholar
- 59.Taowei David Wang and Bijan Parsia. Ontology performance profiling and model examination: First steps. In Karl Aberer, Key-Sun Choi, Natasha Fridman Noy, Dean Allemang, Kyung-Il Lee, Lyndon J. B. Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Mauroux, editors, Proceedings of the 6th International Semantic Web Conference/2nd Asian Semantic Web Conference (ISWC/ASWC’07), volume 4825 of Lecture Notes in Computer Science, pages 595–608, Busan, Korea, November. Springer, Berlin, 2007.Google Scholar
- 60.Taowei David Wang, Bijan Parsia, and James A. Hendler. A survey of the web ontology landscape. In Isabel Cruz, Stefan Decker, Dean Allemang Chris Preist, Daniel Schwabe, Peter Mika, Michael Uschold, and Lora Aroyo, editors, Proceedings of the Fifth International Semantic Web Conference (ISWC’06), volume 4273 of Lecture Notes in Computer Science, pages 682–694, Athens, Georgia, November 2006. Springer.Google Scholar
- 61.Jonathan Yu, James A. Thom, and Audrey Tam. Ontology evaluation using wikipedia categories for browsing. In Proceedings of the ACM Sixteenth Conference on Information and Knowledge Management (CIKM), Lisboa, Portugal, November 2007. ACM.Google Scholar