Skip to main content
SpringerLink
Log in

The Evaluation of Ontologies

Toward Improved Semantic Interoperability

  • Chapter
Semantic Web

Abstract

Recent years have seen rapid progress in the development of ontologies as semantic models intended to capture and represent aspects of the real world. There is, however, great variation in the quality of ontologies. If ontologies are to become progressively better in the future, more rigorously developed, and more appropriately compared, then a systematic discipline of ontology evaluation must be created to ensure quality of content and methodology. Systematic methods for ontology evaluation will take into account representation of individual ontologies, performance (in terms of accuracy, domain coverage and the efficiency and quality of automated reasoning using the ontologies) on tasks for which the ontology is designed and used, degree of alignment with other ontologies and their compatibility with automated reasoning. A sound and systematic approach to ontology evaluation is required to transform ontology engineering into a true scientific and engineering discipline. This chapter discusses issues and problems in ontology evaluation, describes some current strategies, and suggests some approaches that might be useful in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Sean B., van Harmelen F., Hendler J., Horrocks I., McGuinness D.L., Patel-Schneider P.F., and Stein L.A. 2004. OWL Web Ontology Language Reference. W3C Recommendation 10 February 2004. http://www.w3.org/TR/owl-ref/.

    Google Scholar 

  2. Blaschke C., Hirschman L., Valencia A., and Yeh A. A critical assessment of text mining methods in molecular biology. BMC Bioinformatics (22-article special issue), Volume 6, Supplement 1. 2004. http://www.biomedcentral.com/1471-2105/6?issue=Sl.

    Google Scholar 

  3. Bodenreider O., Smith B.; Kumar A, and Burgun A. Investigating subsumption in DL-based terminologies: a case study in SNOMED-CT, in: U. Hahn, S. Schulz and R. Cornet (eds.), Proceedings of the First International Workshop on Formal Biomedical Knowledge Representation (KR-MED 2004), 12–20, 2004

    Google Scholar 

  4. Janez B., Grobelnik M., and Mladenić D. Ontology evaluation, deliverable D1.6.1, EU-IST Project IST-2003-506826 Semantically Enabled Knowledge Technologies (SEKT), Jožef Stefan Institute, Ljubljana, Slovenia, May 8, 2005, 2005a.

    Google Scholar 

  5. Janez B. Grobelnik M., and Mladenić D. A survey of ontology evaluation techniques. SiKDD05. 2005b.

    Google Scholar 

  6. Brewster C, Alani H., Dasmahapatra S. and Wilk Y. Data driven ontology evaluation. In Proceedings of International Conference on Language Resources and Evaluation, Lisbon, Portugal, 2004.

    Google Scholar 

  7. Camon E., Barrell D., Dimmer E., Lee V., Magrane M., Maslen J., Binns D., and Apweiler R. 2005. An evaluation of GO annotation retrieval for BioCreAtlvE and GOA. BMC Bioinformatics 6(1): S17, 2005.

    Article  PubMed  Google Scholar 

  8. Capability Maturity Model Integration (CMMI). Software Engineering Institute, Carnegie-Mellon University. http://www.sei.cmu.edu/cmmi.

    Google Scholar 

  9. Carletta J. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2):249–254, 1996.

    Google Scholar 

  10. Ceusters W., Smith B., and Flanagan J. Ontology and medical terminology: why description logics are not enough, in Proceedings of the Conference: Towards an Electronic Patient Record (TEPR 2003), San Antonio 10–14 May 2003, Boston, MA: Medical Records Institute (CD-ROM publication), 2003.

    Google Scholar 

  11. Ceusters W., Smith B., Kumar A., Dhaen C. Ontology-based error detection in SNOMED-CT, in M. Fieschi, et al. (eds.), Medinfo 2004, Amsterdam: IOS Press, 482–486, 2004a.

    Google Scholar 

  12. Ceusters W., Smith B., and Fielding J.M. LinkSuite: formally robust ontology-based data and information integration. In Rahm E (Ed.): Data Integration in the Life Sciences: DILS 2004, (Lecture Notes in Computer Science 2994) Springer 2004, p. 124–139, 2004b.

    Google Scholar 

  13. Ceusters W and Smith B. A terminological and ontological analysis of the NCI thesaurus. Methods of Information in Medicine 2005; 44: 498–507, 2005.

    CAS  Google Scholar 

  14. Ceusters W and Smith B. A realism-based approach to the evolution of biomedical ontologies. Forthcoming in Proceedings of the AMIA 2006 Annual Symposium, Washington DC, November 11–15, 2006a.

    Google Scholar 

  15. Ceusters W and Smith B. Towards A realism-based metric for quality assurance in ontology matching (forthcoming in Proceedings of FOIS-2006), 2006b.

    Google Scholar 

  16. Cohen P., Schrag R., Jones E., Pease A., Lin A., Starr B., Easter D., Gunning D., and Burke M. The DARPA High Performance Knowledge Bases project. Artificial Intelligence Magazine, vol. 19, no. 4, 25–49, 1998.

    Google Scholar 

  17. Cornet R. and Abu-Hanna A.. Usability of expressive description logics—a case study in UMLS. Proceedings of AMIA Symp 2002: 180–4, 2002.

    Google Scholar 

  18. Common Logic Standard, June 21, 2006 version., http://cl.tamu.edu/docs/cl/24707-21-June-2006.pdf

    Google Scholar 

  19. Daconta M, Smith K., and Obrst L. The Semantic Web: The Future of XML, Web Services, and Knowledge Management. John Wiley, Inc., 2003.

    Google Scholar 

  20. Daude J., Padro L., and Rigau G. A Complete WN1.5 to WN1.6 Mapping. NAACL-2001 Workshop on WordNet and Other Lexical Resources: Applications, Extension, and Customization, 83–88, 2001.

    Google Scholar 

  21. Dellarocas C. Reputation mechanisms. Forthcoming in Handbook on Economics and Information Systems. (T. Hendershott, ed.), Elsevier Publishing. 2006. http://www.rhsmith.umd.edu/faculty/cdell/papers/elsevierchapter.pdf.

    Google Scholar 

  22. Ehrig M. and Sure Y. Ontology mapping-an integrated approach. In Proceedings of the First European Semantic Web Symposium, ESWS 2004, volume 3053 of Lecture Notes in Computer Science, pages 76–91, Heraklion, Greece, Springer Verlag, May 2004.

    Google Scholar 

  23. European Bioinformatics Institute (EBI) Gene Ontology Annotation (GOA) Project. http://www.ebi.ac.uk/GOA/.

    Google Scholar 

  24. Euzénat J., Le Bach L., Barrasa J., Bouquet P., De Bo J., Dieng R. Ehrig M., Hauswirth N., Jarrar M., Lara R., Maynard D., Napoli A., Stamou G., Stuckenschmidt H., Shvaiko P., Tessaris S., Van Acker S., Zaihrayeu I. KnowledgeWeb Deliverable D2.2.3: State of the art on ontology alignment. V1.2, August 2004. http://www.starlab.vub.ac.be/research/projects/knowledgeweb/kweb-223.pdf.

    Google Scholar 

  25. Fernandez M. Overview of methodologies for building ontologies. Workshop on Ontologies and Problem-Solving Methods: Lessons Learned and Future Trends. (IJCAI99). August. 1999. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-18/4-fernandez.pdf.

    Google Scholar 

  26. Fernandez M., Gómez-Pérez A., and Juristo N. METHONTOLOGY: from ontological art to ontological engineering. Workshop on Ontological Engineering. Spring Spring Symposium Series. AAAI97, Stanford, 1997.

    Google Scholar 

  27. Friedland N.S., Allen P.G., Witbrok M., Matthews G., Salay N., Miraglia P., Angele J., Staab S., Israel D., Chaudhri V., Porter B., Barker K., and Clark P. Towards a quantitative, platform-independent analysis of knowledge systems. Proceedings of KR’2004, 2004.

    Google Scholar 

  28. Gangemi A., Catenacci C, Ciaramita, M., and Lehmann J. A theoretical framework for ontology evaluation and validation. In Proceedings of SWAP2005, 2005

    Google Scholar 

  29. Gangemi A., Carola C., Massimiliano C., and Lehmann J. Modelling ontology evaluation and validation. To appear in Proceedings of ESWC2006, Springer, 2006.

    Google Scholar 

  30. Golbeck J., Fragoso G., Hartel F., Hendler J., Oberthaler J., and Parsia B. The National Cancer Institute’s Thesaurus and Ontology. Journal of Web Semantics. 1(1), 2003. http://www.websemanticsjournal.org/ps/pub/2004-6.

    Google Scholar 

  31. Goldberg L.J., Ceusters W., Eisner J., and Smith B. The Significance of SNODENT, Stud Health Technol Inform. 116:737–742, 2005.

    PubMed  Google Scholar 

  32. Grenon P. Spatio-temporality in basic formal ontology: SNAP and SPAN, upperlevel ontology, and framework of formalization (part I). Technical Report Series 05/2003, IFOMIS, 2003.

    Google Scholar 

  33. Gruninger M. and Fox M.S. Methodology for the design and evaluation of ontologies. In: Proceedings of the Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI-95, Montreal, 1995.

    Google Scholar 

  34. Guarino N., and Welty C. Evaluating ontological decisions with OntoClean. Communications of the ACM. 45(2):61–65, 2002. New York: ACM Press. http://portal.acm.org/citation.cfm?doid=503124.503150.

    Google Scholar 

  35. Hahn U. and Schulz S. Towards a broad-coverage biomedical ontology based on description logics. Pacific Symposium on Biocomputing 8, 2003, 577–588, 2003.

    Google Scholar 

  36. Hartmann J., Spyns P., Giboin A., Maynard D., Cuel R., and Suárez-Figueroa M.C. D1.2.3 Methods for ontology evaluation. EU-IST Network of Excellence (NoE) IST-2004-507482 KWEB Deliverable D1.2.3 (WP 1.2), 2005.

    Google Scholar 

  37. IET. RKF Y1 evaluation report, October 2001. http://www.iet.com/Projects/RKF/IET-RKF-Y1-Evaluation.ppt.

    Google Scholar 

  38. Interoperable Knowledge Representation for Intelligence Support (IKRIS). http://nrrc.mitre.org/NRRC/ikris.htm.

    Google Scholar 

  39. ISO/IEC 11179 specification. http://metadata-standards.org/.

    Google Scholar 

  40. Kalfoglou Y. and Schorlemmer M. Ontology mapping: the state of the art. Knowledge Engineering Review, 18(1): 1–31, 2003.

    Article  Google Scholar 

  41. Kashyap V., Ramakrishnan C., Thomas C, and Sheth A. TaxaMiner; an experimental framework for automated taxonomy bootstrapping. International Journal of Web and Grid Services, Special Issue on Semantic Web and Mining Reasoning, September 2005.

    Google Scholar 

  42. Kashyap V. and Borgida A. Representing the UMLS semantic network using OWL: (Or “What’s in a Semantic Web link?”). In: Fensel D, Sycara K, Mylopoulos J, editors. The SemanticWeb — ISWC 2003. Heidelberg: Springer-Verlag, p. 1–16, 2003.

    Google Scholar 

  43. Klein M. Combining and relating ontologies: an analysis of problems and solutions. In A. Gomez-Perez, M. Gruninger, H. Stuckenschmidt, and M. Uschold, editors, Workshop on Ontologies and Information Sharing, IJCAI01, Seattle, USA, 2001.

    Google Scholar 

  44. Köhler J., Munn K., Rüegg A., Skusa A., and Smith B. Quality control for terms and definitions in ontologies and taxonomies, BMC Bioinform, 7:212–220, 2006.

    Article  Google Scholar 

  45. Kumar A. and Smith B. The Unified Medical Language System and the Gene Ontology, KI 2003:;135–148, 2003.

    Google Scholar 

  46. Kumar A. and Smith B. Oncology ontology in the NCI Thesaurus, Artificial Intelligence in Medicine Europe (AIME), (Lecture Notes in Computer Science 3581), 213–220, 2005.

    Google Scholar 

  47. Kumar A. and Smith B. The Ontology of processes and functions: a study of the international classification of functioning, disability and health, 2006. http://ontology.buffalo.edu/medo/ICF.pdf.

    Google Scholar 

  48. Lenat D.B. Cyc: a large-scale investment in knowledge infrastructure. Communications of the ACM 38, no. 11, 1995.

    Google Scholar 

  49. Lewen H., Supekar K., Noy N.F., and Musen M.A. TopicSpecific Trust and Open Rating Systems: An approach for ontology evaluation, Workshop on Evaluation of Ontologies for the Web EON 2006, WWW2006, Edinburgh, UK, May 22, 2006.

    Google Scholar 

  50. Linstone H.A. and Turoff M., Editors. The delphi method: techniques and applications. 2006. http://www.is.njit.edu/pubs/delphibook/. Electronic reproduction of: Linstone, H. & Turoff, M. The Delphi Method: Techniques and Applications. Reading, Ma.: Addison-Wesley, 1975.

    Google Scholar 

  51. Mahesh K., Nirenburg S., and Beale S. KR requirements for natural language semantics: a critical evaluation of Cyc. Proceedings of KR-96, 1996

    Google Scholar 

  52. Mani I., Samuel S., Concepcion K., and Vogel D. Automatically inducing ontologies from corpora. Proceedings of CompuTerm 2004: 3rd International Workshop on Computational Terminology, COLING’2004, Geneva, 2004.

    Google Scholar 

  53. Open Biomedical Ontologies (OBO) Foundry. http://obofoundry.org.

    Google Scholar 

  54. Obrst L., Liu H., Wray R. Ontologies for corporate web applications. Artificial Intelligence Magazine, special issue on Ontologies, American Association for Artificial Intelligence, Chris Welty, ed., Fall, 2003, pp. 49–62, 2003.

    Google Scholar 

  55. Obrst L., Hughes T., Ray S. Prospects and possibilities for ontology evaluation: the view from NCOR. Workshop on Evaluation of Ontologies for the Web (EON2006), Edinburgh, UK, May 22, 2006.

    Google Scholar 

  56. Patel C, Supekar K., Lee Y., and Park E. Ontokhoj: a semantic web portal for ontology searching, ranking and classification. In Proc. of the Fifth ACM Workshop on Web Information and Data Management, pages 58–61, New Orleans, Louisiana, USA, ACM Press, 2003.

    Google Scholar 

  57. Pisanelli D.M., Gangemi A., and Steve G. An ontological analysis of the UMLS Methathesaurus. Proceedings of AMIA Symp. 810–4, 1998.

    Google Scholar 

  58. Polcicová G. and Návrat P. Semantic similarity in content-based filtering: In Proc. of ADBI2002 Advances in Databases and Information Systems, Manolopoulos, Y. and Návrat, P. (Eds.), Springer LNCS 2435, 2002, 80–85, 2002.

    Google Scholar 

  59. Rogers J.E. and Rector A.L. GALEN’s Model of parts and wholes: experience and comparisons Annual Fall Symposium of American Medical Informatics Association, Los Angeles CA Hanley & Belfus Inc Philadelphia PA;:714–8

    Google Scholar 

  60. Sanguino R. Evaluation of Cyc. LEF Grant Report, CSC, Miami, FL, March 2001, http://www2.csc.com/lef/programs/grants/finalpapers/sanguino_eval_cyc.pdf

    Google Scholar 

  61. Seipel D. and Baumeister J. Declarative methods for the evaluation of ontologies. University of Wurzburg. 2004.

    Google Scholar 

  62. Shannon C.E. A mathematical theory of communication, Bell System Technical Journal, vol. 27, pp. 379–423, 623–656, July, October, 1948

    Google Scholar 

  63. Sider T. Four-Dimensionalism: An Ontology of Persistence and Time. Oxford: Oxford University Press, 2002.

    Google Scholar 

  64. Siegel, S. and Castellan N. J. Nonparametric Statistics for the Behavioural Sciences. McGraw-Hill, 2nd edition, 1988.

    Google Scholar 

  65. Simons P. Parts: A Study in Ontology. Clarendon Press, Oxford, 1987.

    Google Scholar 

  66. Smith B., Ceusters W., and Temmerman R. Wüsteria, Stud Health Technol Inform. 2005;116:647–652, 2005.

    PubMed  Google Scholar 

  67. Smith B. and Ceusters W. HL7 RIM: An incoherent standard, Stud Health Technol Inform, in press, 2006.

    Google Scholar 

  68. Smith B. and Kumar A. On controlled vocabularies in bioinformatics: a case study in the Gene Ontology, BIOSILICO: Drug Discovery Today, 2:246–252, 2004.

    Article  CAS  Google Scholar 

  69. Smith B., Williams J., and Schulze-Kremer S. The ontology of the Gene Ontology, Proc AMIA Symp. 609–613, 2003.

    Google Scholar 

  70. Smith B. Against idiosyncracy in ontology develoment, under review. 2006.

    Google Scholar 

  71. Smith B. From concepts to clinical reality: an essay on the benchmarking of biomedical terminologies, J Biomed Inform, 2006; 39(3): 288–298, 2006a.

    Article  PubMed  Google Scholar 

  72. Barry S., Jacob Köhler J., and Kumar A. On the application of formal principles to life science data: a case study in the Gene Ontology. Erhard Rahm (Ed.): Data Integration in the Life Sciences, First International Workshop, DILS 2004, Leipzig, Germany, March 25–26, 2004, Proceedings, pp. 79–94. Lecture Notes in Computer Science 2994 Springer, 2004.

    Google Scholar 

  73. Systematized Nomenclature of Medicine (SNOMED) Clinical Terms (CT). http://www.snomed.org/.

    Google Scholar 

  74. Van Buggenhout C. and Ceusters W. A novel view on information content of concepts in a large ontology and a view on the structure and the quality of the ontology. International Journal of Medical Informatics 74(2–4): 125–32, 2005.

    Article  PubMed  Google Scholar 

  75. Varzi A.C. Basic problems of mereotopology. In: Guarino, N., ed. Formal Ontology in Information Systems, Amsterdam: IOS Press, pp. 29–38, 1998.

    Google Scholar 

  76. Wolstencroft K., McEntire R., Stevens R., Tabernero L. and Brass A. Constructing ontology-driven protein family databases. Bioinformatics 21(8): 1685–1692, 2005.

    Article  PubMed  CAS  Google Scholar 

  77. Wroe C.J., Stevens R., Goble C.A., Ashburner M. A methodology to migrate the Gene Ontology to a description logic environment using DAML+OIL. Pac Symp Biocomput 2003:624–35, 2000.

    Google Scholar 

  78. Extended Metadata Registry (XMDR). http://xmdr.org/.

    Google Scholar 

  79. Zhang S., Bodenreider O., and Golbreich C. Experience in reasoning with the Foundational Model of Anatomy in OWL DL. Pacific Symposium on Biocomputing 2006: World Scientific; 200–211, 2006.

    Google Scholar 

  80. Zhong J., Zhu H., Li J., and Yu Y. Conceptual graph matching for semantic search. In Priss U, Corbett D, Angelova G (eds.) Conceptual Structures: Integration and Interfaces (ICCS2002), 92–106., 2002

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MITRE, USA

    Leo Obrst & Inderjeet Mani

  2. State University of New York at Buffalo, USA

    Werner Ceusters & Barry Smith

  3. US National Institute of Standards and Technology, USA

    Steve Ray

Authors
  1. Leo Obrst
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Werner Ceusters
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Inderjeet Mani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Steve Ray
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Barry Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Knowledge Discovery Department, Institute for Infocomm Research, Singapore

    Christopher J. O. Baker

  2. Center for Medical Informatics, Yale University School of Medicine, New Haven, CT, USA

    Kei-Hoi Cheung

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Obrst, L., Ceusters, W., Mani, I., Ray, S., Smith, B. (2007). The Evaluation of Ontologies. In: Baker, C.J.O., Cheung, KH. (eds) Semantic Web. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-48438-9_8

Download citation

Publish with us

Policies and ethics

Search

Navigation