Optimize First, Buy Later: Analyzing Metrics to Ramp-Up Very Large Knowledge Bases

  • Paea LePendu
  • Natalya F. Noy
  • Clement Jonquet
  • Paul R. Alexander
  • Nigam H. Shah
  • Mark A. Musen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6496)


As knowledge bases move into the landscape of larger ontologies and have terabytes of related data, we must work on optimizing the performance of our tools. We are easily tempted to buy bigger machines or to fill rooms with armies of little ones to address the scalability problem. Yet, careful analysis and evaluation of the characteristics of our data—using metrics—often leads to dramatic improvements in performance. Firstly, are current scalable systems scalable enough? We found that for large or deep ontologies (some as large as 500,000 classes) it is hard to say because benchmarks obscure the load-time costs for materialization. Therefore, to expose those costs, we have synthesized a set of more representative ontologies. Secondly, in designing for scalability, how do we manage knowledge over time? By optimizing for data distribution and ontology evolution, we have reduced the population time, including materialization, for the NCBO Resource Index, a knowledge base of 16.4 billion annotations linking 2.4 million terms from 200 ontologies to 3.5 million data elements, from one week to less than one hour for one of the large datasets on the same machine.


Query Time Load Time Ontology Evolution Biomedical Ontology Materialization Cost 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bernstein, A., Kiefer, C., Stocker, M.: OptARQ: A SPARQL Optimization Approach based on Triple Pattern Selectivity Estimation. Technical report, University of Zurich (2007)Google Scholar
  2. 2.
    Bodenreider, O., Smith, B., Kumar, A., Burgun, A.: Investigating Subsumption in SNOMED CT: An Exploration into Large Description Logic-Based Biomedical Ontologies. Artificial Intelligence in Medicine 39(3), 183–195 (2007)CrossRefGoogle Scholar
  3. 3.
    Dai, M., Shah, N., Xuan, W.: An Efficient Solution for Mapping Free Text to Ontology Terms. In: AMIA Summit on Translational Bioinformatics (2008)Google Scholar
  4. 4.
    d’Aquin, M., Baldassarre, C., Gridinoc, L., Angeletou, S., Sabou, M., Motta, E.: Characterizing Knowledge on the Semantic Web with Watson. In: Evaluation of Ontology-based Tools Wkshp (ISWC), pp. 1–10 (2007)Google Scholar
  5. 5.
    Doms, A., Schroeder, M.: GoPubMed: exploring PubMed with the Gene Ontology. Nucleic Acids Research 33(Web Server issue), 783–786 (2005)CrossRefGoogle Scholar
  6. 6.
    Tummarello, G., Oren, E., Delbru, R.: Weaving the Open Linked Data. In: Int’l Sem. Web Conf. (ISWC), pp. 547–560 (2007)Google Scholar
  7. 7.
    Guo, Y., Pan, Z., Heflin, J.: LUBM: A Benchmark for OWL Knowledge Base Systems. Journal of Web Semantics 3(2), 158–182 (2005)CrossRefGoogle Scholar
  8. 8.
    Horridge, M., Bechhofer, S.: The OWL API: A Java API for Working with OWL 2 Ontologies. In: OWL Experiences and Directions Wkshp, OWLED (2009)Google Scholar
  9. 9.
    Jonquet, C., Musen, M., Shah, N.: Building a Biomedical Ontology Recommender Web Service. Biomedical Semantics 1(S1) (2010)Google Scholar
  10. 10.
    Jonquet, C., Shah, N., Youn, C., Callendar, C., Storey, M., Musen, M.: NCBO Annotator: Semantic Annotation of Biomedical Data. In: Int’l Sem. Web Conf., ISWC (2009)Google Scholar
  11. 11.
    Krallinger, M., Leitner, F., Valencia, A.: Analysis of biological processes and diseases using text mining approaches. In: Methods in Molecular Biology, pp. 341–382 (2010)Google Scholar
  12. 12.
    LePendu, P., Dou, D.: Using Ontology Databases for Scalable Query Answering, Inconsistency Detection, and Data Integration. Intelligent Info. Sys., JIIS (2010)Google Scholar
  13. 13.
    Ma, L., Yang, Y., Qiu, Z., Xie, G., Pan, Y., Liu, S.: Towards a Complete OWL Ontology Benchmark. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 125–139. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Magkanaraki, A., Sofia, A., Christophides, V., Plexousakis, D.: Benchmarking RDF Schemas for the Semantic Web. In: Horrocks, I., Hendler, J. (eds.) ISWC 2002. LNCS, vol. 2342, pp. 132–146. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. 15.
    Marwah, K., Katzin, D., Alterovitz, G.: Context-Specific Ontology Integration. In: (under review) Pacific Symp. on Biocomputing, PSB (2011)Google Scholar
  16. 16.
    Noy, N., Shah, N., Whetzel, P., Dai, B., Dorf, M., Griffith, N., Jonquet, C., Rubin, D., Storey, M.A., Chute, C., Musen, M.: BioPortal: Ontologies and Integrated Data Resources at the Click of a Mouse. Nucleic Acids Research 1(37) (2009)Google Scholar
  17. 17.
    Pan, Z., Heflin, J.: DLDB: Extending Relational Databases to Support Semantic Web Queries. In: Practical and Scalable Sem. Web Systems Wkshp (ISWC), pp. 109–113 (2003)Google Scholar
  18. 18.
    Rubin, D., Supekar, K., Mongkolwat, P., Kleper, V., Channin, D.: Annotation and Image Markup: Accessing and Interoperating with the Semantic Content in Medical Imaging. IEEE Intelligent Systems 24(1), 57–65 (2009)CrossRefGoogle Scholar
  19. 19.
    Shah, N., Jonquet, C., Chiang, A., Butte, A., Chen, R., Musen, M.: Ontology-driven Indexing of Public Datasets for Translational Bioinformatics. BMC Bioinformatics 10 (2009)Google Scholar
  20. 20.
    Stenson, P., Ball, E., Howells, K., Phillips, A., Mort, M., Cooper, D.: The Human Gene Mutation Database: providing a comprehensive central mutation database for molecular diagnostics and personalized genomics. Hum. Genomics 4(2), 69–72 (2009)CrossRefGoogle Scholar
  21. 21.
    Tempich, C., Volz, R.: Towards a benchmark for Semantic Web reasoners - an analysis of the DAML ontology library. In: Evaluation of Ontology-based Tools Wkshp, ISWC (2003)Google Scholar
  22. 22.
    Tirrell, R., Evani, U., Berman, A., Mooney, S., Musen, M., Shah, N.: An Ontology-Neutral Framework for Enrichment Analysis. In: AMIA Annual Symposium (2010)Google Scholar
  23. 23.
    Urbani, J., Kotoulas, S., Oren, E., Harmelen, F.: Scalable Distributed Reasoning using MapReduce. In: Bernstein, A., Karger, D.R., Heath, T., Feigenbaum, L., Maynard, D., Motta, E., Thirunarayan, K. (eds.) ISWC 2009. LNCS, vol. 5823, pp. 634–649. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  24. 24.
    Wang, S., Guo, Y., Qasem, A., Heflin, J.: Rapid Benchmarking for Semantic Web Knowledge Base Systems. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (eds.) ISWC 2005. LNCS, vol. 3729, pp. 758–772. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  25. 25.
    Wang, T., Parsia, B., Hendler, J.: A Survey of the Web Ontology Landscape. In: Int’l Sem. Web Conf. (ISWC), pp. 682–694 (2009)Google Scholar
  26. 26.
    Weaver, J., Hendler, J.: Parallel Materialization of the Finite RDFS Closure for Hundreds of Millions of Triples. In: Bernstein, A., Karger, D.R., Heath, T., Feigenbaum, L., Maynard, D., Motta, E., Thirunarayan, K. (eds.) ISWC 2009. LNCS, vol. 5823, pp. 682–697. Springer, Heidelberg (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Paea LePendu
    • 1
  • Natalya F. Noy
    • 1
  • Clement Jonquet
    • 1
  • Paul R. Alexander
    • 1
  • Nigam H. Shah
    • 1
  • Mark A. Musen
    • 1
  1. 1.Stanford UniversityStanfordUSA

Personalised recommendations