Exploring Comparative Evaluation of Semantic Enrichment Tools for Cultural Heritage Metadata

  • Hugo Manguinhas
  • Nuno Freire
  • Antoine IsaacEmail author
  • Juliane Stiller
  • Valentine Charles
  • Aitor Soroa
  • Rainer Simon
  • Vladimir Alexiev
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9819)


Semantic enrichment of metadata is an important and difficult problem for digital heritage efforts such as Europeana. This paper gives motivations and presents the work of a recently completed Task Force that addressed the topic of evaluation of semantic enrichment. We especially report on the design and the results of a comparative evaluation experiment, where we have assessed the enrichments of seven tools (or configurations thereof) on a sample benchmark dataset from Europeana.


Semantic enrichment Metadata Cultural heritage Evaluation Europeana 


  1. 1.
    Bunescu, R., Paşca, M.: Using encyclopedic knowledge for named entity disambiguation. In: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 9–16 (2006)Google Scholar
  2. 2.
    Stiller, J., Petras, V., Gäde, M., Isaac, A.: Automatic enrichments with controlled vocabularies in Europeana: challenges and consequences. In: Ioannides, M., Magnenat-Thalmann, N., Fink, E., Žarnić, R., Yen, A.-Y., Quak, E. (eds.) EuroMed 2014. LNCS, vol. 8740, pp. 238–247. Springer, Heidelberg (2014)Google Scholar
  3. 3.
    Agirre, E., Barrena, A., Lopez de Lacalle, O., Soroa A., Fernando S., Stevenson, M.: Matching cultural heritage items to Wikipedia. In: Proceedings of LREC 2012, Istanbul, Turkey (2012)Google Scholar
  4. 4.
    Isaac, A., Manguinhas, H., Stiller, J., Charles, V.: Report on Enrichment and Evaluation. The Hague, Netherlands (2015).
  5. 5.
    Usbeck, R., Röder, M., Ngonga Ngomo, A., et al.: GERBIL: general entity annotator benchmarking framework. In: Proceedings of 24th WWW Conference. ACM (2015).
  6. 6.
    Griffiths, J., Basset, S., Goodale, P., et al.: Evaluation of the second PATHS prototype. Technical report, Paths Project (2014)Google Scholar
  7. 7.
    Olensky, M., Stiller, J., Dröge, E.: Poisonous India or the importance of a semantic and multilingual enrichment strategy. In: Dodero, J.M., Palomo-Duarte, M., Karampiperis, P. (eds.) MTSR 2012. CCIS, vol. 343, pp. 252–263. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Monti, J., Monteleone, M., Buono, M., et al: Cross-lingual information retrieval and semantic interoperability for cultural heritage repositories. In: RANLP 2013. Hissar, Bulgaria (2013).
  9. 9.
    Petras, V., et al.: Cultural heritage in CLEF (CHiC) 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 192–211. Springer, Heidelberg (2013)Google Scholar
  10. 10.
    Dobreva, M., Chowdhury, S.: A user-centric evaluation of the Europeana digital library. In: Chowdhury, G., Koo, C., Hunter, J. (eds.) ICADL 2010. LNCS, vol. 6102, pp. 148–157. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Tordai, A., van Ossenburg, J., Schreiber, G., et al.: Let’s agree to disagree: on the evaluation of vocabulary alignment. In: Proceedings of 6th K-CAP. ACM (2011)Google Scholar
  12. 12.
    Isaac, A., Manguinhas, H., Charles, V., Stiller, J., et al.: Comparative evaluation of semantic enrichments. Technical report (2015). Report available at Data archive available at:
  13. 13.
    Freire, N., Borbinha, J., Calado, P., Martins, B.: A metadata geoparsing system for place name recognition and resolution in metadata records. In: ACM/IEEE Joint Conference on Digital Libraries (2011).
  14. 14.
    Charles, V., Freire, N., Antoine, I.: Links, languages and semantics: linked data approaches in the European Library and Europeana. In: Linked Data in Libraries: Let’s make it happen! IFLA 2014, Satellite Meeting on Linked Data in Libraries (2014)Google Scholar
  15. 15.
    Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley Series in Probability and Mathematical Statistics, pp. 212–236. John Wiley & Sons Ltd (1981). (Chap. 13)Google Scholar
  16. 16.
    Randolph, J.: Free-marginal multirater Kappa (multirater κ free): an alternative to fleiss’ fixed-marginal multirater Kappa. Joensuu University Learning and Instruction Symposium 2005, Joensuu, Finland, 14–15 October 2005.
  17. 17.
    Manning, C., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008).

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Hugo Manguinhas
    • 1
  • Nuno Freire
    • 1
  • Antoine Isaac
    • 1
    • 6
    Email author
  • Juliane Stiller
    • 2
  • Valentine Charles
    • 1
  • Aitor Soroa
    • 3
  • Rainer Simon
    • 4
  • Vladimir Alexiev
    • 5
  1. 1.Europeana FoundationThe HagueThe Netherlands
  2. 2.Humboldt-Universität zu BerlinBerlinGermany
  3. 3.University of the Basque CountryBizkaiaSpain
  4. 4.Austrian Institute of TechnologyViennaAustria
  5. 5.Ontotext CorpSofiaBulgaria
  6. 6.Vrije Universiteit AmsterdamAmsterdamThe Netherlands

Personalised recommendations