A Decade of Evaluating Europeana - Constructs, Contexts, Methods & Criteria

  • Vivien PetrasEmail author
  • Juliane Stiller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10450)


This meta-analysis of 41 evaluation studies of the Europeana Digital Library categorizes them by their constructs, contexts, criteria, and methodologies using Saracevic’s digital library evaluation framework. The analysis shows that system-centered evaluations prevail over user-centered evaluations and evaluations from a societal or institutional perspective are missing. The study reveals, which Europeana components have received focused attention in the last decade (e.g. the metadata) and can serve as a reference for identifying gaps, selecting methodologies and re-using data for future evaluations.


Digital library evaluation Evaluation constructs Europeana Meta-analysis 


  1. 1.
    Akasereh, M., Malak, P., Pawłowski, A.: Evaluation of IR strategies for Polish. In: Przepiórkowski, A., Ogrodniczuk, M. (eds.) NLP 2014. LNCS, vol. 8686, pp. 384–391. Springer, Cham (2014). doi: 10.1007/978-3-319-10888-9_38 Google Scholar
  2. 2.
    van den Akker, C., van Nuland, A., van der Meij, L., van Erp, M., Legêne, S., Aroyo, L., Schreiber, G.: From information delivery to interpretation support: evaluating cultural heritage access on the web. In: WebSci 2013, pp. 431–440. ACM (2013)Google Scholar
  3. 3.
    Aletras, N., Stevenson, M.: Computing similarity between cultural heritage items using multimodal features. In: LaTeCH 2012, pp. 85–93. ACL (2012)Google Scholar
  4. 4.
    Aletras, N., Stevenson, M., Clough, P.: Computing similarity between items in a digital library of cultural heritage. J. Comp. Cult. Heri. 5(4), 16:1–16:19 (2012)Google Scholar
  5. 5.
    Aloia, N., Concordia, C., Gerwen, A.M., Hansen, P., Kuwahara, M., Ly, A.T., Meghini, C., Spyratos, N., Sugibuchi, T., Tanaka, Y., Yang, J., Zeni, N.: Design, implementation and evaluation of a user generated content service for Europeana. In: Gradmann, S., Borri, F., Meghini, C., Schuldt, H. (eds.) TPDL 2011. LNCS, vol. 6966, pp. 477–482. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-24469-8_54 CrossRefGoogle Scholar
  6. 6.
    Candela, L., Castelli, D., Ferro, et al.: The DELOS digital library reference model. Version 0.96. Technical report, DELOS Network of Excellence on Digital Libraries (2007)Google Scholar
  7. 7.
    Ceccarelli, D., Gordea, S., Lucchese, C., Nardini, F.M., Tolomei, G.: Improving Europeana search experience using query logs. In: Gradmann, S., Borri, F., Meghini, C., Schuldt, H. (eds.) TPDL 2011. LNCS, vol. 6966, pp. 384–395. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-24469-8_39 CrossRefGoogle Scholar
  8. 8.
    CIBER Research Ltd.: Europeana 2012–2013: usage and performance update. Technical report, CIBER Research Ltd. (2013).
  9. 9.
    Clark, D., Nicholas, I., Rowlands, I.: D3.1.3 Report on best-practice and how users are using the Europeana service. Technical report, D 3.1.3, EuropeanaConnect (2011)Google Scholar
  10. 10.
    Clough, P., Otegi, A., Agirre, E., Hall, M.: Implementing recommendations in the PATHS system. In: Bolikowski, Ł., Casarosa, V., Goodale, P., Houssos, N., Manghi, P., Schirrwagen, J. (eds.) TPDL 2013. CCIS, vol. 416, pp. 169–173. Springer, Cham (2014). doi: 10.1007/978-3-319-08425-1_17 Google Scholar
  11. 11.
    Şencan, I.: The performance evaluation of the information retrieval system of the Europeana website. In: IMCW 2013, p. 59 (2013)Google Scholar
  12. 12.
    Dangerfield, M.C., Kalshoven, L.: Report and recommendations from the task force on metadata quality. Technical report, Europeana (2015).
  13. 13.
    Dani, A., Chatzopoulou, C., Siatri, R., Mystakopoulos, F., Antonopoulou, S., Katrinaki, E., Garoufallou, E.: Digital libraries evaluation: measuring Europeana’s usability. In: Garoufallou, E., Hartley, R.J., Gaitanou, P. (eds.) MTSR 2015. CCIS, vol. 544, pp. 225–236. Springer, Cham (2015). doi: 10.1007/978-3-319-24129-6_20 CrossRefGoogle Scholar
  14. 14.
    Dickel, J.: Digitale Bibliotheken im Vergleich: Europeana & WDL. Perspektive Bibliothek 4(1), 45–67 (2015)Google Scholar
  15. 15.
    Dobreva, M., Chowdhury, S.: A user-centric evaluation of the Europeana digital library. In: Chowdhury, G., Koo, C., Hunter, J. (eds.) ICADL 2010. LNCS, vol. 6102, pp. 148–157. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-13654-2_19 CrossRefGoogle Scholar
  16. 16.
    Dobreva, M., McCulloch, E., Birrell, D., Ünal, Y., Feliciati, P.: Digital natives and specialised digital libraries: a study of Europeana users. In: Kurbanoğlu, S., Al, U., Lepon Erdoğan, P., Tonta, Y., Uçak, N. (eds.) IMCW 2010. CCIS, vol. 96, pp. 45–60. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-16032-5_5 CrossRefGoogle Scholar
  17. 17.
    Fox, E.A., Gonçalves, M.A., Shen, R.: Theoretical foundations for digital libraries: the 5s (societies, scenarios, spaces, structures, streams) approach. Synth. Lect. Inform. Concepts, Retrieval, Serv. 4(2), 1–180 (2012)CrossRefzbMATHGoogle Scholar
  18. 18.
    Fuhr, N., Hansen, P., Mabe, M., Micsik, A., Sølvberg, I.: Digital libraries: a generic classification and evaluation scheme. In: Constantopoulos, P., Sølvberg, I.T. (eds.) ECDL 2001. LNCS, vol. 2163, pp. 187–199. Springer, Heidelberg (2001). doi: 10.1007/3-540-44796-2_17 CrossRefGoogle Scholar
  19. 19.
    Fuhr, N., Tsakonas, G., Aalberg, T., et al.: Evaluation of digital libraries. IJDL 8(1), 21–38 (2007)Google Scholar
  20. 20.
    Gäde, M.: Country and language level differences in multilingual digital libraries. Dissertation, Humboldt-Universität zu Berlin, April 2014Google Scholar
  21. 21.
    Gäde, M., Petras, V.: Multilingual interface preferences. In: IIiX 2014, pp. 231–234. ACM (2014)Google Scholar
  22. 22.
    Glaser, B.G., Strauss, A.: The discovery of ground theory (1967)Google Scholar
  23. 23.
    Gonçalves, M.A., Fox, E.A., Watson, L.T., Kipp, N.A.: Streams, structures, spaces, scenarios, societies (5s): a formal model for digital libraries. ACM TOIS 22(2), 270–312 (2004)CrossRefGoogle Scholar
  24. 24.
    Gonçalves, M.A., Moreira, B.L., Fox, E.A., Watson, L.T.: “What is a good digital library?” - a quality model for digital libraries. IPM 43(5), 1416–1437 (2007)Google Scholar
  25. 25.
    Gonzalez-Agirre, A., Rigau, G., Agirre, E., Aletras, N., Stevenson, M.: Why are these similar? Investigating item similarity types in a large digital library. JASIST 67(7), 1624–1638 (2016)Google Scholar
  26. 26.
    Griffiths, J., et al.: D5.2 evaluation of the second PATHS prototype. Technical report (2014).
  27. 27.
    Hill, T., Charles, V., Stiller, J., Isaac, A.: Searching for inspiration: user needs and search architecture in Europeana collections. In: ASIS&T, pp. 1–7. Wiley (2016)Google Scholar
  28. 28.
    Isaac, A., Manguinhas, H., Stiller, J., Charles, V.: Final report on evaluation and enrichment. Technical report, Europeana (2015).
  29. 29.
    Joo, S., Xie, I.: Evaluation constructs and criteria for digital libraries: a document analysis. In: Recent Developments in the Design, Construction, and Evaluation of Digital Libraries: Case Studies, pp. 126–140. IGI Global (2013)Google Scholar
  30. 30.
    Kapidakis, S.: Comparing metadata quality in the Europeana context. In: PETRA 2012, p. 25:1–25:8. ACM (2012)Google Scholar
  31. 31.
    Király, P.: A metadata quality assurance framework. Technical report, Europeana (2015).
  32. 32.
    Manguinhas, H., Freire, N., Isaac, A., Stiller, J., Charles, V., Soroa, A., Simon, R., Alexiev, V.: Exploring comparative evaluation of semantic enrichment tools for cultural heritage metadata. In: Fuhr, N., Kovács, L., Risse, T., Nejdl, W. (eds.) TPDL 2016. LNCS, vol. 9819, pp. 266–278. Springer, Cham (2016). doi: 10.1007/978-3-319-43997-6_21 CrossRefGoogle Scholar
  33. 33.
    Marchionini, G.: Evaluating digital libraries: a longitudinal and multifaceted view. Libr. Trends 49(2), 304 (2000)Google Scholar
  34. 34.
    Moreira, B.L., Gonçalves, M.A., Laender, A.H., Fox, E.A.: Automatic evaluation of digital libraries with 5squal. J. Informetrics 3(2), 102–123 (2009)CrossRefGoogle Scholar
  35. 35.
    Navarrete, T.: Europeana as online cultural information service. Study Report (2016).
  36. 36.
    Nicholas, D., Clark, D.: The second digital transition: to the mobile space - an analysis of Europeana. Learn. Publishing 26(4), 240–252 (2013)CrossRefGoogle Scholar
  37. 37.
    Nicholas, D., Clark, D., Rowlands, I., Jamali, H.R.: Information on the Go: a case study of Europeana mobile users. JASIST 64(7), 1311–1322 (2013)CrossRefGoogle Scholar
  38. 38.
    Olensky, M., Stiller, J., Dröge, E.: Poisonous India or the importance of a semantic and multilingual enrichment strategy. In: Dodero, J.M., Palomo-Duarte, M., Karampiperis, P. (eds.) MTSR 2012. CCIS, vol. 343, pp. 252–263. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-35233-1_25 CrossRefGoogle Scholar
  39. 39.
    Peters, C., Thanos, C. (eds.): ECDL 1997. LNCS, vol. 1324. Springer, Heidelberg (1997)Google Scholar
  40. 40.
    Petras, V., Bogers, T., Toms, E., Hall, M., Savoy, J., Malak, P., Pawłowski, A., Ferro, N., Masiero, I.: Cultural heritage in CLEF (CHiC) 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 192–211. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40802-1_23 CrossRefGoogle Scholar
  41. 41.
    Petras, V., Ferro, N., Gäde, M., Isaac, A., Kleineberg, M., Masiero, I., Nicchio, M., Stiller, J.: Cultural Heritage in CLEF (CHiC) Overview 2012. In: CLEF (2012)Google Scholar
  42. 42.
    Petras, V., Hill, T., Stiller, J., Gäde, M.: Europeana - a search engine for digitised cultural heritage material. Datenbank-Spektrum 17(1), 41–46 (2017)CrossRefGoogle Scholar
  43. 43.
    Purday, J.: Think culture: from concept to construction. Electron. Libr. 27(6), 919–937 (2009)CrossRefGoogle Scholar
  44. 44.
    Saracevic, T.: Digital library evaluation: toward an evolution of concepts. Libr. Trends 49(2), 350–369 (2000)Google Scholar
  45. 45.
    Saracevic, T.: Evaluation of digital libraries: an overview. In: Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004)Google Scholar
  46. 46.
    Schindler, A., Gordea, S., Biessum, H.: The Europeana sounds music information retrieval pilot. In: Ioannides, M., Fink, E., Moropoulou, A., Hagedorn-Saupe, M., Fresa, A., Liestøl, G., Rajcic, V., Grussenmeyer, P. (eds.) EuroMed 2016. LNCS, vol. 10059, pp. 109–117. Springer, Cham (2016). doi: 10.1007/978-3-319-48974-2_13 CrossRefGoogle Scholar
  47. 47.
    Schlötterer, J., Seifert, C., Wagner, L., Granitzer, M.: A game with a purpose to access europe’s cultural treasure. ECIR 2015, 13–18 (2015)Google Scholar
  48. 48.
    Schweibenz, W.: Eine erste evaluation der Europeana: Wie Benutzer das look & feel des prototypen der Europäischen digitalen bibliothek beurteilen. Inf. Wissenschaft und Praxis 61(5), 277–284 (2010)Google Scholar
  49. 49.
    Stiller, J., Gäde, M., Petras, V.: Multilingual access to digital libraries: the Europeana use case. Inf.-Wissenschaft und Praxis 64, 86–95 (2013)Google Scholar
  50. 50.
    Stiller, J.: From curation to collaboration. Dissertation, Humboldt-Universität zu Berlin, Philosophische Fakultät I (2014)Google Scholar
  51. 51.
    Stiller, J., Isaac, A., Petras, V.: EuropeanaTech task force on a multilingual and semantic enrichment strategy. Technical report, Europeana (2014).
  52. 52.
    Stiller, J., Olensky, M., Petras, V.: A framework for the evaluation of automatic metadata enrichments. In: Closs, S., Studer, R., Garoufallou, E., Sicilia, M.-A. (eds.) MTSR 2014. CCIS, vol. 478, pp. 238–249. Springer, Cham (2014). doi: 10.1007/978-3-319-13674-5_23 Google Scholar
  53. 53.
    Stiller, J., Petras, V.: A framework for classifying and comparing interactions in cultural heritage information systems. In: Cultural Heritage Information: Access and management. Facet Publishing (2015)Google Scholar
  54. 54.
    Stiller, J., Petras, V., Gäde, M., Isaac, A.: Automatic enrichments with controlled vocabularies in Europeana: challenges and consequences. In: Ioannides, M., Magnenat-Thalmann, N., Fink, E., Žarnić, R., Yen, A.-Y., Quak, E. (eds.) EuroMed 2014. LNCS, vol. 8740, pp. 238–247. Springer, Cham (2014). doi: 10.1007/978-3-319-13695-0_23 Google Scholar
  55. 55.
    Sykes, J., Dobreva, M., Birrell, D., McCulloch, E., Ruthven, I., Ünal, Y., Feliciati, P.: A new focus on end users: eye-tracking analysis for digital libraries. In: Lalmas, M., Jose, J., Rauber, A., Sebastiani, F., Frommholz, I. (eds.) ECDL 2010. LNCS, vol. 6273, pp. 510–513. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15464-5_69 CrossRefGoogle Scholar
  56. 56.
    Tsakonas, G., Mitrelis, A., Papachristopoulos, L., Papatheodorou, C.: An exploration of the digital library evaluation literature based on an ontological representation. JASIST 64(9), 1914–1926 (2013)CrossRefGoogle Scholar
  57. 57.
    Tsakonas, G., Papatheodorou, C.: An ontological representation of the digital library evaluation domain. JASIST 62(8), 1577–1593 (2011)CrossRefGoogle Scholar
  58. 58.
    Valtysson, B.: Europeana - the digital construction of Europe’s collective memory. Inf. Commun. Soc. 15(2), 151–170 (2012)CrossRefGoogle Scholar
  59. 59.
    Xie, H.: Evaluation of digital libraries: criteria and problems from users’ perspectives. Libr. Inf. Sci. Res. 28(3), 433–452 (2006)CrossRefGoogle Scholar
  60. 60.
    Xie, H.I.: Users’ evaluation of digital libraries (DLs): their uses, their criteria, and their assessment. IPM 44(3), 1346–1373 (2008)Google Scholar
  61. 61.
    Xie, I., Matusiak, K.K.: Chapter 10 - evaluation of digital libraries. In: Discover Digital Libraries, pp. 281–318. Elsevier, Oxford (2016)Google Scholar
  62. 62.
    Yankova, I., Velev, E., Nusheva, K., Sonia, S.: The European digital library-factor for long-life learning in arts and cultural studies. Qual. Quant. Methods Libr. 4(4), 965–971 (2015)Google Scholar
  63. 63.
    Zhang, Y.: Developing a holistic model for digital library evaluation. JASIST 61(1), 88–110 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Berlin School of Library and Information ScienceHumboldt-Universität zu BerlinBerlinGermany

Personalised recommendations