Evaluation of Matching Systems

  • Jérôme Euzenat
  • Pavel Shvaiko

Abstract

The increasing number of methods available for ontology matching commands the evaluation of these methods.

Matching systems are difficult to compare, but the ontology matching field can only evolve if evaluation criteria are provided. These should help system designers assess the strengths and weaknesses of their systems as well as help application developers choose the most appropriate algorithm.

References

  1. Aguirre, J.L., Eckert, K., Euzenat, J., Ferrara, A., van Hage, W.R., Hollink, L., Meilicke, C., Nikolov, A., Ritze, D., Scharffe, F., Shvaiko, P., Šváb-Zamazal, O., Trojahn, C., Jiménez-Ruiz, E., Cuenca Grau, B., Zapilko, B.: Results of the ontology alignment evaluation initiative 2012. In: Proc. 7th International Workshop on Ontology Matching (OM) at the 11th International Semantic Web Conference (ISWC), Boston, MA, USA, pp. 73–115 (2012) Google Scholar
  2. Alexe, B., Tan, W.C., Velegrakis, Y.: Comparing and evaluating mapping systems with STBenchmark. Proc. VLDB Endow. 1(2), 1468–1471 (2008b) Google Scholar
  3. Alexe, B., Tan, W.C., Velegrakis, Y.: STBenchmark: towards a benchmark for mapping systems. Proc. VLDB Endow. 1(1), 230–244 (2008c) Google Scholar
  4. Avesani, P., Giunchiglia, F., Yatskevich, M.: A large scale taxonomy mapping evaluation. In: Proc. 4th International Semantic Web Conference (ISWC), Galway, Ireland. Lecture Notes in Computer Science, vol. 3729, pp. 67–81 (2005) Google Scholar
  5. David, J., Euzenat, J.: On fixing semantic alignment evaluation measures. In: Proc. 3rd International Workshop on Ontology Matching (OM) at the 7th International Semantic Web Conference (ISWC), Karlsruhe, Germany, pp. 25–36 (2008) Google Scholar
  6. Do, H.-H., Melnik, S., Rahm, E.: Comparison of schema matching evaluations. In: Proc. Workshop on Web, Web-Services, and Database Systems, Erfurt, Germany. Lecture Notes in Computer Science, vol. 2593, pp. 221–237 (2002) CrossRefGoogle Scholar
  7. Duchateau, F., Bellahsene, Z., Hunt, E.: XBenchMatch: a benchmark for XML schema matching tools. In: Proc. 33rd Very Large Databases (VLDB) Demo Track, Wien, Austria, pp. 1318–1321 (2007a) Google Scholar
  8. Ehrig, M.: Ontology Alignment: Bridging the Semantic Gap. Springer, New York (2007). 0-387-32805-X Google Scholar
  9. Ehrig, M., Euzenat, J.: Relaxed precision and recall for ontology matching. In: Proc. International Workshop on Integrating Ontologies at the 3rd International Conference on Knowledge Capture (K-CAP), Banff, Canada, pp. 25–32 (2005) Google Scholar
  10. Ehrig, M., Staab, S.: QOM—quick ontology mapping. In: Proc. 3rd International Semantic Web Conference (ISWC), Hiroshima, Japan. Lecture Notes in Computer Science, vol. 3298, pp. 683–697 (2004) Google Scholar
  11. Euzenat, J.: Towards composing and benchmarking ontology alignments. In: Proc. International Workshop on Semantic Integration at the 2nd International Semantic Web Conference (ISWC), Sanibel Island, FL, USA, pp. 165–166 (2003) Google Scholar
  12. Euzenat, J.: Semantic precision and recall for ontology alignment evaluation. In: Proc. 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, pp. 348–353 (2007) Google Scholar
  13. Euzenat, J., Isaac, A., Meilicke, C., Shvaiko, P., Stuckenschmidt, H., Šváb, O., Svátek, V., van Hage, W.R., Yatskevich, M.: Results of the ontology alignment evaluation initiative 2007. In: Proc. 2nd International Workshop on Ontology Matching (OM) at the 6th International Semantic Web Conference (ISWC) and the 2nd Asian Semantic Web Conference (ASWC), Busan, Korea, pp. 96–132 (2007a) Google Scholar
  14. Euzenat, J., Meilicke, C., Shvaiko, P., Stuckenschmidt, H., Trojahn, C.: Ontology Alignment Evaluation Initiative: six years of experience. J. Data Semant. XV, 158–192 (2011) CrossRefGoogle Scholar
  15. Euzenat, J., Roşoiu, M., Trojahn, C.: Ontology matching benchmarks: generation, stability, and discriminability. J. Web Semant. 21, 30–48 (2013) CrossRefGoogle Scholar
  16. Ferrara, A., Lorusso, D., Montanelli, S., Varese, G.: Towards a benchmark for instance matching. In: Proc. 3rd International Workshop on Ontology Matching (OM) at the 7th International Semantic Web Conference (ISWC), Karlsruhe, Germany, pp. 37–48 (2008) Google Scholar
  17. Ferrara, A., Montanelli, S., Noessner, J., Stuckenschmidt, H.: Benchmarking matching applications on the semantic web. In: Proc. 8th Extended Semantic Web Conference (ESWC), Hersounisous, Greece. Lecture Notes in Computer Science, vol. 6644, pp. 108–122 (2011a) Google Scholar
  18. Fleischhacker, D., Stuckenschmidt, H.: A practical implementation of semantic precision and recall. In: Proc. 4th International Conference on Complex, Intelligent and Software Intensive Systems (CISIS), Krakow, Poland, pp. 986–991 (2010) CrossRefGoogle Scholar
  19. García Castro, R., Maynard, D., Foxvog, D., Wache, H., González-Cabero, R.: Specification of a methodology, general criteria, and benchmark suites for benchmarking ontology tools. Deliverable D2.1.4, Knowledge web NoE (2004) Google Scholar
  20. Giunchiglia, F., Yatskevich, M., Avesani, P., Shvaiko, P.: A large scale dataset for the evaluation of ontology matching systems. Knowl. Eng. Rev. 24(2), 137–157 (2009) CrossRefGoogle Scholar
  21. Hammer, J., Stonebraker, M., Topsakal, O.: THALIA: Test Harness for the Assessment of Legacy Information Integration Approaches. In: Proc. 21st International Conference on Data Engineering (ICDE), Tokyo, Japan, pp. 485–486 (2005) CrossRefGoogle Scholar
  22. He, B., Chang, K.: Automatic complex schema matching across web query interfaces: a correlation mining approach. ACM Trans. Database Syst. 31(1), 1–45 (2006) CrossRefGoogle Scholar
  23. Isaac, A., Matthezing, H., van der Meij, L., Schlobach, S., Wang, S., Zinn, C.: Putting ontology alignment in context: usage scenarios, deployment and evaluation in a library case. In: Proc. 5th European Semantic Web Conference (ESWC), Tenerife, Spain. Lecture Notes in Computer Science, vol. 5021, pp. 402–417 (2008) Google Scholar
  24. Isaac, A., Wang, S., Zinn, C., Matthezing, H., van der Meij, L., Schlobach, S.: Evaluating thesaurus alignments for semantic interoperability in the library domain. IEEE Intell. Syst. 24(2), 76–86 (2009) CrossRefGoogle Scholar
  25. Langlais, P., Véronis, J., Simard, M.: Methods and practical issues in evaluating alignment techniques. In: Proc. 17th International Conference on Computational Linguistics (CoLing), Montréal, Canada, pp. 711–717 (1998) CrossRefGoogle Scholar
  26. Marie, A., Gal, A.: Boosting schema matchers. In: Proc. 16th International Conference on Cooperative Information Systems (CoopIS), Monterrey, Mexico. Lecture Notes in Computer Science, vol. 5331, pp. 283–300 (2008) Google Scholar
  27. Meilicke, C., Garcia Castro, R., Freitas, F., van Hage, W.R., Montiel Ponsoda, E., Ribeiro de Azevedo, R., Stuckenschmidt, H., Šváb-Zamazal, O., Svátek, V., Tamilin, A., Trojahn, C., Wang, S.: MultiFarm: a benchmark for multilingual ontology matching. J. Web Semant. 15, 62–68 (2012) CrossRefGoogle Scholar
  28. Melnik, S., Garcia Molina, H., Rahm, E.: Similarity flooding: a versatile graph matching algorithm. In: Proc. 18th International Conference on Data Engineering (ICDE), San Jose, CA, USA, pp. 117–128 (2002) CrossRefGoogle Scholar
  29. Niu, X., Wang, H., Wu, G., Qi, G., Yu, Y.: Evaluating the stability and credibility of ontology matching methods. In: Proc. 8th Extended Semantic Web Conference (ESWC), Hersounisous, Greece. Lecture Notes in Computer Science, vol. 6643, pp. 275–289 (2011) Google Scholar
  30. Noy, N., Musen, M.: Evaluating ontology-mapping tools: requirements and experience. In: Proc. 1st International Workshop on Evaluation of Ontology Tools (EON) at the 13th International Conference on Knowledge Engineering and Knowledge Management (EKAW), Siguenza, Spain, pp. 1–14 (2002a) Google Scholar
  31. Paulheim, H., Hertling, S., Ritze, D.: Towards evaluating interactive ontology matching tools. In: Proc. 10th European Semantic Web Conference (ESWC), Montpellier, France, pp. 31–45 (2013) Google Scholar
  32. Porzel, R., Malaka, R.: A task-based approach for ontology evaluation. In: Proc. Workshop on Ontology Learning and Population at the 16th Eureopean Conference on Artificial Intelligence (ECAI), Valencia, Spain (2004) Google Scholar
  33. Su, W., Wang, J., Lochovsky, F.: Holistic schema matching for web query interfaces. In: Proc. 10th Conference on Extending Database Technology (EDBT), Munich, Germany. Lecture Notes in Computer Science, vol. 3896, pp. 77–94 (2006) Google Scholar
  34. Sun, A., Lin, E.-P.: Hierarchical text classification and evaluation. In: Proc. 1st International Conference on Data Mining (ICDM), San Jose, CA, USA, pp. 521–528 (2001) Google Scholar
  35. Tordai, A., van Ossenbruggen, J., Wielinga, B.: Let’s agree to disagree: on the evaluation of vocabulary alignment. In: Proc. 6th International Conference on Knowledge Capture (K-CAP), Banff, Canada, pp. 65–72 (2011) Google Scholar
  36. Tournaire, R.: Découverte automatique de correspondances entre ontologies. PhD thesis, Université de Grenoble, Grenoble, France (2010) Google Scholar
  37. Trojahn, C., Meilicke, C., Euzenat, J., Stuckenschmidt, H.: Automating OAEI campaigns (first report). In: Proc. International Workshop on Evaluation of Semantic Technologies (iWEST) at the 9th International Semantic Web Conference (ISWC), Shanghai, China (2010a) Google Scholar
  38. Vaccari, L., Shvaiko, P., Pane, J., Besana, P., Marchese, M.: An evaluation of ontology matching in geo-service applications. GeoInformatica 16(1), 31–66 (2012) CrossRefGoogle Scholar
  39. van Hage, W.R.: Evaluating ontology alignment techniques. PhD thesis, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands (2007) Google Scholar
  40. van Rijsbergen, C.J.K.: Information Retrieval. Butterworths, London (1975). http://www.dcs.gla.ac.uk/Keith/Preface.html Google Scholar
  41. Zaiss, K., Conrad, S., Vater, S.: A benchmark for testing instance-based ontology matching methods. In: Proc. 17th International Conference on Knowledge Engineering and Knowledge Management (EKAW), Posters and Demos, Lisbon, Portugal (2010) Google Scholar
  42. Zhang, S., Mork, P., Bodenreider, O.: Lessons learned from aligning two representations of anatomy. In: Proc. 13th International Conference on the Principles of Knowledge Representation and Reasoning Conference (KR), Whistler, Canada, pp. 555–560 (2004) Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Jérôme Euzenat
    • 1
  • Pavel Shvaiko
    • 2
  1. 1.INRIA and LIGGrenobleFrance
  2. 2.Informatica Trentina SpA, while at Department of Engineering and Computer Science (DISI), University of Trento, while at Web of Data, Bruno Kessler Foundation - IRSTTrentoItaly

Personalised recommendations