Evaluation Measures for Ontology Matchers in Supervised Matching Scenarios

  • Dominique Ritze
  • Heiko Paulheim
  • Kai Eckert
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8219)


Precision and Recall, as well as their combination in terms of F-Measure, are widely used measures in computer science and generally applied to evaluate the overall performance of ontology matchers in fully automatic, unsupervised scenarios. In this paper, we investigate the case of supervised matching, where automatically created ontology alignments are verified by an expert. We motivate and describe this use case and its characteristics and discuss why traditional, F-measure based evaluation measures are not suitable for this use case. Therefore, we investigate several alternative evaluation measures and propose the use of Precision@N curves as a mean to assess different matching systems for supervised matching. We compare the ranking of several state of the art matchers using Precision@N curves to the traditional F-measure based ranking, and discuss means to combine matchers in a way that optimizes the user support in supervised ontology matching.


Supervised Ontology Matching Evaluation Recall Precision F-Measure Precision@N-Curves ROC-Curves Precision-Recall-Curves 


  1. 1.
    Aguirre, J.L., Eckert, K., Euzenat, J., Ferrara, A., van Hage, W.R., Hollink, L., Meilicke, C., Nikolov, A., Ritze, D., Scharffe, F., Shvaiko, P., Šváb-Zamazal, O., Trojahn, C., Jiménez-Ruiz, E., Cuenca Grau, B., Zapilko, B.: Results of the Ontology Alignment Evaluation Initiative 2012. In: Proc. of the 7th Int. Workshop on Ontology Matching (2012)Google Scholar
  2. 2.
    Ba, M., Diallo, G.: ServOMap and ServOMap-lt Results for OAEI 2012. In: Proc. of the 7th ISWC Workshop on Ontology Matching, OM (2012) W.R Hage, L. Hollink, C. Meilicke, A. Nikolov, D. Ritze, F. Scharffe, P. Shvaiko, O.Š. Zamazal, C. Trojahn, E. Jim/’enez-Ruiz, B.C. Grau, and B. ZapilkoGoogle Scholar
  3. 3.
    Cheatham, M.: MapSSS Results for OAEI 2011. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)Google Scholar
  4. 4.
    Dang, T.T., Gabriel, A., Hertling, S., Roskosch, P., Wlotzka, M., Zilke, J.R., Janssen, F., Paulheim, H.: HotMatch Results for OEAI 2012. In: Proc.of the 7th ISWC Workshop on Ontology Matching, OM (2012)Google Scholar
  5. 5.
    David, J., Guillet, F., Briand, H.: Association rule ontology matching approach. International Journal on Semantic Web and Information Systems 3(2), 27–49 (2007)CrossRefGoogle Scholar
  6. 6.
    Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. In: Proc. of the 23rd Int. Conference on Machine Learning, ICML (2006)Google Scholar
  7. 7.
    Duan, S., Fokoue, A., Srinivas, K.: One Size Does Not Fit All: Customizing Ontology Alignment Using User Feedback. In: Patel-Schneider, P.F., Pan, Y., Hitzler, P., Mika, P., Zhang, L., Pan, J.Z., Horrocks, I., Glimm, B. (eds.) ISWC 2010, Part I. LNCS, vol. 6496, pp. 177–192. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Eckert, K., Meilicke, C., Stuckenschmidt, H.: Improving Ontology Matching using Meta-level Learning. In: Aroyo, L., et al. (eds.) ESWC 2009. LNCS, vol. 5554, pp. 158–172. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  9. 9.
    Ehrig, M., Staab, S., Sure, Y.: Bootstrapping Ontology Alignment Methods with APFEL. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (eds.) ISWC 2005. LNCS, vol. 3729, pp. 186–200. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Euzenat, J., Shvaiko, P.: Ontology Matching. Springer (2007)Google Scholar
  11. 11.
    Fangmeyer, H.: Semi Automatic Indexing: State of the Art. Technical report, North Atlantic Treaty Organization. Advisory Group for Aerospace Research and Development (1974)Google Scholar
  12. 12.
    Gay, C.W., Kayaalp, M., Aronson, A.R.: Semi-Automatic Indexing of Full Text Biomedical Articles. In: Proc. of the Fall Symposium of the American Medical Informatics Association (2005)Google Scholar
  13. 13.
    Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29–36 (1982)Google Scholar
  14. 14.
    Hazewinkel, M.: Encyclopedia of Mathematics. Harmonic series. Springer (2001)Google Scholar
  15. 15.
    Hertling, S.: Hertuda Results for OEAI 2012. In: Proc. of the 7th ISWC Workshop on Ontology Matching, OM (2012)Google Scholar
  16. 16.
    Huber, J., Sztyler, T., Noessner, J., Meilicke, C.: CODI: Combinatorial Optimization for Data Integration - Results for OAEI 2011. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)Google Scholar
  17. 17.
    Jiménez-Ruiz, E., Grau, B.C., Zhou, Y., Horrocks, I.: Large-scale interactive ontology matching: Algorithms and implementation. In: Proc. of the 20th European Conference on Artificial Intelligence, ECAI (2012)Google Scholar
  18. 18.
    Kirsten, T., Gross, A., Hartung, M., Rahm, E.: Gomma: a component-based infrastructure for managing and analyzing life science ontologies and their evolution. Journal of Biomedical Semantics 2(6) (2011)Google Scholar
  19. 19.
    Manning, C., Schütze, H.: Foundations of statistical natural language processing. MIT Press (1999)Google Scholar
  20. 20.
    Mayr, P., Petras, V.: Building a Terminology Network for Search: The KoMoHe Project. In: Proc. of the Int. Conference on Dublin Core and Metadata Applications (2008)Google Scholar
  21. 21.
    Ngo, D., Bellahsene, Z.: YAM++: A Multi-strategy Based Approach for Ontology Matching Task. In: ten Teije, A., Völker, J., Handschuh, S., Stuckenschmidt, H., d’Acquin, M., Nikolov, A., Aussenac-Gilles, N., Hernandez, N. (eds.) EKAW 2012. LNCS, vol. 7603, pp. 421–425. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  22. 22.
    Ngomo, A.-C.N., Jens, L., Auer, S., Höffner, K.: RAVEN - active learning of link specifications. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)Google Scholar
  23. 23.
    Niu, X., Wang, H., Wu, G., Qi, G., Yu, Y.: Evaluating the Stability and Credibility of Ontology Matching Methods. In: Antoniou, G., Grobelnik, M., Simperl, E., Parsia, B., Plexousakis, D., De Leenheer, P., Pan, J. (eds.) ESWC 2011, Part I. LNCS, vol. 6643, pp. 275–289. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  24. 24.
    Noy, N.F., Musen, M.A.: The PROMPT suite: interactive tools for ontology merging and mapping. Int. Journal of Human-Computer Studies 59(6), 983–1024 (2003)CrossRefGoogle Scholar
  25. 25.
    Paulheim, H.: WeSeE-Match Results for OEAI 2012. In: Proc. of the 7th ISWC Workshop on Ontology Matching, OM (2012)Google Scholar
  26. 26.
    Paulheim, H., Hertling, S., Ritze, D.: Towards evaluating interactive ontology matching tools. In: Cimiano, P., Corcho, O., Presutti, V., Hollink, L., Rudolph, S. (eds.) ESWC 2013. LNCS, vol. 7882, pp. 31–45. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  27. 27.
    Ritze, D., Paulheim, H.: Towards an Automatic Parameterization of Ontology Matching Tools based on Example Mappings. In: Proc. of the 6th ISWC Workshop on Ontology Matching, OM (2011)Google Scholar
  28. 28.
    Thomas Jr., G.B., Finney, R.L.: Calculus and Analytic Geometry, 9th edn. Addison Wesley (1996)Google Scholar
  29. 29.
    To, H.-V., Ichise, R., Le, H.-B.: An Adaptive Machine Learning Framework with User Interaction for Ontology Matching. In: Proc. of the International Joint Conferences on Artifical Intelligence, Workshop on Information Integration on the Web, pp. 35–40 (2009)Google Scholar
  30. 30.
    van Rijsbergen, C.J.: Information retrieval. Butterworths, London (1975)Google Scholar
  31. 31.
    Zhou, H., Kang, J., Chen, F., Yang, H.: OPTIMA: An Ontology-Based PlaTform-specIfic software Migration Approach. In: Proc. of the 7th Int. Conference on Quality Software (QSIC), pp. 143–152 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Dominique Ritze
    • 1
  • Heiko Paulheim
    • 1
  • Kai Eckert
    • 1
  1. 1.Research Group Data and Web ScienceUniversity of MannheimMannheimGermany

Personalised recommendations