Ontology Evaluation through Text Classification

  • Yael Netzer
  • David Gabay
  • Meni Adler
  • Yoav Goldberg
  • Michael Elhadad
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5731)


We present a new method to evaluate a search ontology, which relies on mapping ontology instances to textual documents. On the basis of this mapping, we evaluate the adequacy of ontology relations by measuring their classification potential over the textual documents. This data-driven method provides concrete feedback to ontology maintainers and a quantitative estimation of the functional adequacy of the ontology relations towards search experience improvement. We specifically evaluate whether an ontology relation can help a semantic search engine support exploratory search.

We test this ontology evaluation method on an ontology in the Movies domain, that has been acquired semi-automatically from the integration of multiple semi-structured and textual data sources (e.g., IMDb and Wikipedia). We automatically construct a domain corpus from a set of movie instances by crawling the Web for movie reviews (both professional and user reviews). The 1-1 relation between textual documents (reviews) and movie instances in the ontology enables us to translate ontology relations into text classes. We verify that the text classifiers induced by key ontology relations (genre, keywords, actors) achieve high performance and exploit the properties of the learned text classifiers to provide concrete feedback on the ontology.

The proposed ontology evaluation method is general and relies on the possibility to automatically align textual documents to ontology instances.


Textual Document Ontology Relation Movie Review Exploratory Search User Review 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Burkhardt, F., Gulla, J.A., Liu, J., Weiss, C., Zhou, J.: Semi automatic ontology engineering in business applications. In: Proceedings of the 3rd International AST Workshop – Applications of Semantic Technologies. LNI, vol. 134, pp. 688–693 (2008)Google Scholar
  2. 2.
    Baeza-Yates, R., Ciaramita, M., Mika, P., Zaragoza, H.: Towards semantic search. In: Kapetanios, E., Sugumaran, V., Spiliopoulou, M. (eds.) NLDB 2008. LNCS, vol. 5039, pp. 4–11. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  3. 3.
    Gulla, J.A., Borch, H.O., Ingvaldsen, J.E.: Ontology learning for search applications. In: Meersman, R., Tari, Z. (eds.) OTM 2007, Part I. LNCS, vol. 4803, pp. 1050–1062. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: Modelling ontology evaluation and validation. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 140–154. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  5. 5.
    Gomez-Perez, A.: Evaluation of ontologies. International Journal of Intelligent Systems 16, 391–409 (2001)CrossRefzbMATHGoogle Scholar
  6. 6.
    Alani, H., Brewster, C.: Ontology ranking based on the analysis of concept sructures. In: Proceedings of the 3rd International Conference on Knowledge Capture (K-Cap), Banff, Canada, pp. 51–58 (2005)Google Scholar
  7. 7.
    JupiterResearch: Search technology buyerś guide. Technical report, IBM Content Discovery (2006),
  8. 8.
    Aula, A.: Query formulation in web information search. In: Proceedings of IADIS International Conference WWW/Internet, pp. 403–410 (2003)Google Scholar
  9. 9.
    Strasunskas, D., Tomassen, S.L.: Empirical insights on a value of ontology quality in ontology-driven web search. In: Meersman, R., Tari, Z. (eds.) OTM 2008, Part II. LNCS, vol. 5332, pp. 1319–1337. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  10. 10.
    Tartir, S., Arpinar, I., Moore, M., Sheth, A., Aleman-Meza, B.: OntoQA: Metric-based ontology quality analysis. In: Proceedings of Workshop on Knowledge Acquisition, Autonomous, Semantically Heterogeneous Data and Knowledge Sources, pp. 45–53 (2005)Google Scholar
  11. 11.
    White, R.W., Muresan, G., Marchionini, G. (eds.): ACM SIGIR Workshop on Evaluating Exploratory Search Systems, Seattle (2006)Google Scholar
  12. 12.
    Dumais, S., Platt, J., Heckerman, D., Sahami, M.: Inductive learning algorithms and representations for text categorization. In: Proceedings of the Seventh international Conference on Information and Knowledge Management, Bethesda, Maryland, pp. 2–7 (1998)Google Scholar
  13. 13.
    Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  14. 14.
    Akbani, R., Kwek, S.S., Japkowicz, N.: Applying support vector machines to imbalanced datasets. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 39–50. Springer, Heidelberg (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Yael Netzer
    • 1
  • David Gabay
    • 1
  • Meni Adler
    • 1
  • Yoav Goldberg
    • 1
  • Michael Elhadad
    • 1
  1. 1.Department of Computer ScienceBen Gurion University of the NegevBe’er ShevaIsrael

Personalised recommendations