Towards a Complete OWL Ontology Benchmark

  • Li Ma
  • Yang Yang
  • Zhaoming Qiu
  • Guotong Xie
  • Yue Pan
  • Shengping Liu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4011)


Aiming to build a complete benchmark for better evaluation of existing ontology systems, we extend the well-known Lehigh University Benchmark in terms of inference and scalability testing. The extended benchmark, named University Ontology Benchmark (UOBM), includes both OWL Lite and OWL DL ontologies covering a complete set of OWL Lite and DL constructs, respectively. We also add necessary properties to construct effective instance links and improve instance generation methods to make the scalability testing more convincing. Several well-known ontology systems are evaluated on the extended benchmark and detailed discussions on both existing ontology systems and future benchmark development are presented.


Description Logic Query Time Query Response Time Real Ontology Ontology System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Berners-Lee, T., Hendler, J., Lassila, O.: The Semantic WEB. Scientific American (2001)Google Scholar
  2. 2.
    Davies, J., Fensel, D., Harmelen, F. (eds.): Towards the Semantic WEB: Ontology-driven Knowledge Management. John Wiley & Sons Ltd., England (2002)Google Scholar
  3. 3.
    Hayes, P.: Resource Description Framework (RDF): Semantics, W3C Recommendation (2004),
  4. 4.
    Smith, M.K., Welty, C., McGuinness, D.L.: OWL Web Ontology language Guide (2004),
  5. 5.
    Patel-Schneider, P.F., Hayes, P., Horrocks, I.: OWL Web Ontology Language Semantics and Abstract Syntax (2004),
  6. 6.
    Carey, M.J., DeWitt, D.J., Naughtor, J.F.: The 007 Benchmark. In: Proc. of ACM international conference on Management of data, vol. 22(2) (1993) Google Scholar
  7. 7.
    TPC Database Benchmark (2004),
  8. 8.
    Pan, Z., Heflin, J.: DLDB: Extending Relational Databases to Support Semantic Web Queries. In: Proc. of Workshop on Practical and Scaleable Semantic Web Systems, pp. 109–113 (2003)Google Scholar
  9. 9.
    Guo, Y., Pan, Z., Heflin, J.: An Evaluation of Knowledge Base Systems for Large OWL Datasets. In: Proc. of Third International Semantic Web Conference, pp. 274–288 (2004)Google Scholar
  10. 10.
    The Lehigh University Benchmark (2004),
  11. 11.
  12. 12.
    IBM Integrated Ontology Development Toolkit – Minerva (2005),
  13. 13.
    Kopena, J.B., Regli, W.C.: DAMLJessKB: A Tool for Reasoning with the Semantic Web. In: Fensel, D., Sycara, K.P., Mylopoulos, J. (eds.) ISWC 2003. LNCS, vol. 2870, pp. 628–643. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  14. 14.
    Broekstra, J., Kampman, A.: Sesame: A Generic Architecture for Storing and Querying RDF and RDF Schema. In: Horrocks, I., Hendler, J. (eds.) ISWC 2002. LNCS, vol. 2342, p. 54. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. 15.
    Sesame. An Open Source RDF Database with Support for RDF Schema Inferencing and Querying (2002),
  16. 16.
    Tempich, C., Volz, R.: Towards a benchmark for Semantic Web reasoners–an analysis of the DAML ontology library. In: Workshop on Evaluation on Ontology-based Tools, ISWC 2003 (2003)Google Scholar
  17. 17.
    Volz, R.: Web Ontology Reasoning with Logic Databases. Ph.D thesis, AIFB, Karlsruhe (2004) Google Scholar
  18. 18.
    Elhaik, Q., Rousset, M.C., Ycart, B.: Generating Random Benchmarks for Description Logics. In: Proc. of DL 1998 (1998)Google Scholar
  19. 19.
    Horrocks, I., Patel-Schneider, P.: DL Systems Comparison. In: Proc. of DL 1998 (1998)Google Scholar
  20. 20.
    Wang, S., Guo, Y., Qasem, A., Heflin, J.: Rapid Benchmarking for Semantic Web KnowledgeBase Systems, Lehigh University Technical Report LU-CSE-05-026 (2005) Google Scholar
  21. 21.
    OWLIM, OWL Semantic Repository (2005),
  22. 22.
  23. 23.
  24. 24.
    Fikes, R., Hayes, P., Horrocks, I.: OWL-QL - a language for deductive query answering on the Semantic Web. J. of Web Semantics (2004) Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Li Ma
    • 1
  • Yang Yang
    • 1
  • Zhaoming Qiu
    • 1
  • Guotong Xie
    • 1
  • Yue Pan
    • 1
  • Shengping Liu
    • 1
  1. 1.IBM China Research LaboratoryBeijingP.R. China

Personalised recommendations