International Journal on Digital Libraries

, Volume 8, Issue 1, pp 21–38 | Cite as

Evaluation of digital libraries

  • Norbert Fuhr
  • Giannis Tsakonas
  • Trond Aalberg
  • Maristella Agosti
  • Preben Hansen
  • Sarantos Kapidakis
  • Claus-Peter KlasEmail author
  • László Kovács
  • Monica Landoni
  • András Micsik
  • Christos Papatheodorou
  • Carol Peters
  • Ingeborg Sølvberg
Regular Paper


Digital libraries (DLs) are new and innovative information systems, under constant development and change, and therefore evaluation is of critical importance to ensure not only their correct evolution but also their acceptance by the user and application communities. The Evaluation activity of the DELOS Network of Excellence has performed a large-scale survey of current DL evaluation activities. This study has resulted in a description of the state of the art in the field, which is presented in this paper. The paper also proposes a new framework for the evaluation of DLs, as well as for recording, describing and analyzing the related research field. The framework includes a methodology for the classification of current evaluation procedures. The objective is to provide a set of flexible and adaptable guidelines for DL evaluation.


Digital Library Evaluation Framework Information Society Technology Open Archival Information System Information Access System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Fuhr, N., Hansen, P., Mabe, M., Micsik, A., Solvberg, I.: Digital Libraries: A Generic Classification and Evaluation Scheme. In: Lecture Notes In Computer Science, vol. 2163, pp. 187–199. Springer, Berlin (2001) Research and Advanced Technology for Digital Libraries: 5th European Conference, ECDL 2001, Darmstadt, Germany, September 4–9, 2001. ProceedingsGoogle Scholar
  2. 2.
    DELOS: DELOS Workshop on Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics. Technical report (2002) Scholar
  3. 3.
    Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, (2004).Google Scholar
  4. 4.
    Cleverdon, C.: The Cranfield tests on index language devices. In: Sparck-Jones, K., Willett, P. (eds.) Readings in Information Retrieval, pp. 47–59 Morgan Kaufmann, San Francisco (1997)Google Scholar
  5. 5.
    Turpin, A., Hersh, W.: Why batch and user evaluations do not give the same results. In: Proceedings of the SIGIR01, pp. 225–231. ACM, New York (2001)Google Scholar
  6. 6.
    CCSDS: A reference model for an open archival information system. Document Number: ISO 14721:2003 (2003) CCSDS-650.0-B- 1.pdfGoogle Scholar
  7. 7.
    IFLANET: Functional requirements for bibliographic records (1998) Scholar
  8. 8.
    Levy D. and Marshall C. (1995). Going digital: a look at assumptions underlying digital libraries.. Commun. ACM 38: 77–84 CrossRefGoogle Scholar
  9. 9.
    Borgman, C.: What are digital libraries? Competing visions. Inf. Process. Manage. 35, 277–243 (1999)Google Scholar
  10. 10.
    DELOS: Reference models for digital libraries: actors and roles - final report. Technical report (2003) http://www.dli2. actors_final_report.htmlGoogle Scholar
  11. 11.
    Agosti, M., Candela, L., Castelli, D., Ferro, N., Ioannidis Y., Koutrika, G., Meghini, C., Pagano, P., Ross, S., Schek, H.J., Schuldt, H.: A reference model for DLMSs interim report. In: Candela, L., Castelli, D. (eds.) Deliverable D1.4.2 – Reference Model for Digital Library Management Systems [Draft 1], DELOS, A Network of Excellence on Digital Libraries – IST-2002-, Technology-enhanced Learning and Access to Cultural Heritage – delos/2006_WP1_D142/content/pdf?version=1 [last visited 2006, October 2] (2006)Google Scholar
  12. 12.
    Goncalves M.A., Fox E., Kipp N. and Watson L. (2004). Streams, structures, spaces, scenarios, societies (5S): a formal model for digital libraries. ACM Trans. Inf. Syst. 22: 270–312 CrossRefGoogle Scholar
  13. 13.
    Baker, S., Lancaster, F.: The measurement and evaluation of library services. Information Resources Press, Arlington (1991)Google Scholar
  14. 14.
    Nicholson S. (2004). A conceptual framework for the holistic measurement and cumulative evaluation of library services. J. Doc. 60: 164–182 CrossRefGoogle Scholar
  15. 15.
    Kyrillidou, M., Lorie, M., Hobbs, B., Choudhury, G., Webster, D., Flores, N., Heath, F.: Emerging tools for evaluating DL services: conceptual adaptations of LIBQUAL+ and CAPM. J. Digit. Inf. 4, (2003) Articles/v04/i02/HeathGoogle Scholar
  16. 16.
    Kyrillidou, M., Giersch, S.: Developing the digiqual protocol for digital library evaluation. In: JCDL ’05: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries, New York, NY, USA, pp. 172–173. ACM, New York (2005)Google Scholar
  17. 17.
    Tsakonas, G., Kapidakis, S., Papatheodorou, C.: Evaluation of user interaction in digital libraries. In: Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004) http://dlib. Scholar
  18. 18.
    Tsakonas, G., Papatheodorou, C.: Analyzing and evaluating usefulness and usability in electronic information services. J. Inf. Sci. 32(5), 400–419 (2006)Google Scholar
  19. 19.
    Marchionini, G.: Evaluating digital libraries: a longitudinal and multifaceted view. Library Trends 49, (2000)Google Scholar
  20. 20.
    DELOS: A Network of Excellence on Digital Libraries (2004) Technical Annex 1, pag. 6 - DELOS Network of Excellence on Digital Libraries, as part of the Information Society Technologies (IST) Program of the European Commission (Contract G038-507618)Google Scholar
  21. 21.
    ISO: ISO 9241-11. Ergonomic Requirements for Office Work With Visual Display Terminals (VDT). Part 11: Guidance in Usability. International Standards Organization, London (1997)Google Scholar
  22. 22.
    DeGroot, S.P., Knapp, A.E.: Applying the user-centered design (ucd) process to the development of a large bibliographic navigation tool: a partnership between librarian, researcher and developer. Scopus White Paper Series (2004) Scholar
  23. 23.
    Blandford A., Buchanan G. and Jones M. (2004). Usability of digital libraries. Int. J. Digit. Libr. 4: 69–70 CrossRefGoogle Scholar
  24. 24.
    Dickstein, R., Mills, V.: Usability testing at the University of Arizona Library: how to let the users in on the design. Inf. Technol. Libr. 19, (2000)Google Scholar
  25. 25.
    Hilbert D.M. and Redmiles D.F. (2000). Extracting usability information from user interface events. ACM Comput. Surv. 32: 384–421 CrossRefGoogle Scholar
  26. 26.
    Ivory M. and Hearst M. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33: 470–516 CrossRefGoogle Scholar
  27. 27.
    Ke H.R., Kwakkelaar R., Tai Y.M. and Chen L.C. (2002). Exploring behavior of E-journal users in science and technology: transaction log analysis of Elsevier’s ScienceDirect OnSite in Taiwan. Libr. Inf. Sci. Res. 24: 265–291 CrossRefGoogle Scholar
  28. 28.
    Sfakakis, M., Kapidakis, S.: User behavior tendencies on data collections in a digital library. In: Lecture Notes in Computer Science, vol. 2458, pp. 550—559. Springer, Berlin (2002) Research and Advanced Technology for Digital Libraries: 6th European Conference, ECDL 2002, Rome, Italy, September 16–18, 2002. ProceedingsGoogle Scholar
  29. 29.
    Jones S., Cunningham S.J., McNab R. and Boddie S. (2000). A transaction log analysis of a digital library. Int. J. Digit. Libr. 3: 152–169 CrossRefGoogle Scholar
  30. 30.
    Sutcliffe A.G., Ennis M. and Hu J. (2000). Evaluating the effectiveness of visual user interfaces for information retrieval. Int. J. Human-Comput. Stud. 53: 741–763 zbMATHCrossRefGoogle Scholar
  31. 31.
    Kengeri R., Fox E.A., Reddy H.P., Harley H.D. and Seals C.D. (1999). Usability study of digital libraries: ACM, IEEE-CS NCSTRL, NDLTD. Int. J. Digit. Libr. 2: 157–169 CrossRefGoogle Scholar
  32. 32.
    Clark J.A. (2004). A usability study of the Belgian American Research Collection: measuring the functionality of a digital library. OCLC Syst. Serv. Int. Digit. Libr. Perspect. 20: 115–127 Google Scholar
  33. 33.
    Rousseau G.K., Rogers W., Mead S.E., Sit R.A. and Rousseau J.B.A. (1998). Assessing the usability of on-line library systems. Behav. Inf. Technol. 17: 274–281 CrossRefGoogle Scholar
  34. 34.
    Thong J.Y.L., Hong W. and Tam K. (2002). Understanding user acceptance of digital libraries: what are the roles of interface characteristics, organizational context, and individual differences? Int. J. Human-Comput. Stud. 57: 215–242 CrossRefGoogle Scholar
  35. 35.
    Hartson R.H., Shivakumar P. and Perez-Quinones M.A. (2004). Usability inspection of digital libraries: a case study. Int. J. Digit. Libr. 4: 108–123 CrossRefGoogle Scholar
  36. 36.
    Glosiene, A., Manzuch, Z.: Usability of ICT-based Systems: State-of-the-art Review. Technical Report CALIMERA Deliverable 9 (2004)Google Scholar
  37. 37.
    Dicks, R.S.: Mis-usability: on the uses and misuses of usability testing. In: Proceedings of the 20th Annual International Conference on Computer Documentation, New York, pp. 26–30 ACM, New York (2002)Google Scholar
  38. 38.
    Doubleday, A., Springett, M.V., Sutcliffe, A.G., Ryan, M.: A comparison of usability techniques for evaluating design. In: Proceedings of the Conference on “Designing Interactive Systems: Processes, Practices, Methods and Techniques”, New York, pp. 101–110 ACM, New York (1997)Google Scholar
  39. 39.
    Hill L.L., Carver L., Smith T.R., Frew J., Larsgaard M., Dolin R. and Rae M.A. (2000). Alexandria digital library: user evaluation studies and system design. J. Am. Soc. Inf. Sci. 51: 246–259 CrossRefGoogle Scholar
  40. 40.
    Bishop A.P., Merkel C., Neumann L.J., Star S.L., Sandusky R.J. and Ignacio E. (2002). Digital libraries: situating use in changing information infrastructure. J. Am. Soc. Inf. Sci. 51: 394–413 CrossRefGoogle Scholar
  41. 41.
    Fox E.A., Heath L.S., Rao D., Brueni D.J., Nowell L.T., Wake W.C. and Hix D. (1993). Users, user interfaces, and objects: envision, a digital library. J. Am. Soc. Inf. Sci. 44: 480–491 CrossRefGoogle Scholar
  42. 42.
    Buttenfield, B.: Digital libraries: philosophies, technical design considerations, and example scenarios. In: Usability Evaluation of Digital Libraries, pp. 39–59. Haworth Press, New York (1999)Google Scholar
  43. 43.
    Blandford, A., Stelmaszewska, H., Bryann-Kinns, N.: Use of multiple digital libraries: a case study. In: Proceedings of the first ACM/IEEE-CS joint conference on digital libraries, pp. 179—188. ACM Press (2001)Google Scholar
  44. 44.
    Castillo, J., Hartson, H.R., Hix, D.: Remote usability evaluation: can users report their own critical incidents. In: CHI 1998 Conference Summary on Human Factors in Computing Systems, New York, pp. 253–254 ACM, New York (1998)Google Scholar
  45. 45.
    Bates M. (1990). Where should the person stop and the information search interface start? Inf. Process. Manage. 26: 575–591 CrossRefGoogle Scholar
  46. 46.
    Cousins, S.B.: A task-oriented interface to a DL. In: Tauber, M.J. (ed.) Conference Companion on Human Factors in Computing Systems: Common Ground, New York, pp. 103–104. ACM, New York (1996)Google Scholar
  47. 47.
    Pejtersen, A., Fidel, R.: A framework for work centered evaluation and design: a case study of IR on the web. Technical report (1998)Google Scholar
  48. 48.
    Rasmussen, J., Pejtersen, A., Goodstein, L.: Cognitive Systems Engineering. Whiley, New York (1994)Google Scholar
  49. 49.
    Borgman, C.: From Gutenberg to the global information infrastructure: Access to information in the networked World. MIT, Cambridge (2000)Google Scholar
  50. 50.
    Tenopir, C.: Use and users of electronic library resources: an overview and analysis of recent research studies. Council on Library and Information Resources, Washington (2003)Google Scholar
  51. 51.
    Banwell L., Ray K., Coulson G., Urquhart C., Lonsdale R., Armstrong C., Thomas R., Spink S., Yeoman A., Fenton R. and Rowley J. (2004). The JISC user behaviour monitoring and evaluation framework. J. Document. 60: 302–320 CrossRefGoogle Scholar
  52. 52.
    Payette S.D. and Rieger O.Y. (1998). Supporting scholarly inquiry: incorporating users in the design of the digital library. J. Acad. Libr. 24: 121–129 CrossRefGoogle Scholar
  53. 53.
    Yu L. and Apps A. (2000). Studying E-journal user behavior using log files: the experience of superjournal. Libr. Inf. Sci. Res. 22: 311–338 CrossRefGoogle Scholar
  54. 54.
    Wilson T.D. (1997). Information behaviour: an interdisciplinary perspective. Inf. Process. Manage. 33: 551–572 CrossRefGoogle Scholar
  55. 55.
    Kuhlthau C.C. (1991). Inside the search process: information seeking from the user’s perspective. J. Am. Soc. Inf. Sci. 42: 361–371 CrossRefGoogle Scholar
  56. 56.
    Ellis D. and Haugan M. (1997). Modeling the information seeking patterns of engineers and research scientists in an industrial environment. J. Document. 53: 384–403 CrossRefGoogle Scholar
  57. 57.
    Foster A. and Ford N. (2003). Serendipity and information seeking: an empirical study. J. Document. 59: 321–340 CrossRefGoogle Scholar
  58. 58.
    Belkin N.J., Oddy R.N. and Brooks H.M. (1982). ASK for information retrieval: part 1 Background and theory. J. Document. 38: 61–71 Google Scholar
  59. 59.
    Wilson, T.D.: Exploring models of information behaviour: the uncertainty project. Inf. Process. Manage. 893–849 (1999)Google Scholar
  60. 60.
    Wood R. (1996). The Conspectus: a collection analysis and development success. Libr. Acquis. Pract. Theory 20: 429–453 CrossRefGoogle Scholar
  61. 61.
    Saracevic, T.: Evaluation of digital libraries: an overview. In: Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004) Scholar
  62. 62.
    Agosti, M., Ferro, N., Frommholz, I., Thiel, U.: Annotations in digital libraries and collaboratories: facets, models and usage. In: Lecture Notes In Computer Science, vol. 3232, pp. 244—255 Springer, Berlin (2004) Research and Advanced Technology for Digital Libraries: 8th European Conference, ECDL 2004. ProceedingsGoogle Scholar
  63. 63.
    Agosti, M., Albrechtsen, H., Ferro, N., Frommholz, I., Hansen, P., Orio, N., Panizzi, E., Pejtersen, A.M., Thiel, U.: Dilas: a digital library annotation service. In: International Workshop on Annotation for Collaboration. Methods, Tools, and Practices, pp. 91–101 (2005)Google Scholar
  64. 64.
    Hansen, P., Jarvelin, K.: The information seeking and retrieval process at the swedish patent and registration office: moving from lab-based to real life work task environment. In: Agosti, M., Fuhr, N. (eds.) Proceedings of the ACM-SIGIR 2000 Workshop on Patent Retrieval, Athens, Greece (2000) 43–53∼preben/papers/ SIGIR20000-WS.pdfGoogle Scholar
  65. 65.
    Covey, D.: Usage and usability assessment: library practices and concerns, vol. 105. CLIR (2002) Scholar
  66. 66.
    Borgman, C.L.: Social Aspects of Digital Libraries-Final Report to the NSF. Technical report (1996) Scholar
  67. 67.
    Kovács, L., Micsik, A.: The evaluation computer: a model for structuring evaluation activities. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries, Padova, Italy (2004) workshop2004_program.htmlGoogle Scholar
  68. 68.
    Banwell, L., Coulson, G.: Users and user study methodology: the JUBILEE project. Inf. Res. 9, 47–56 (2004) http:// Scholar
  69. 69.
    Jeng J. (2005). What is usability in the context of the digital library and how can it be measured? Inf. Technol. Libr. 24: 47–56 Google Scholar
  70. 70.
    Xie H. (2006). Evaluation of digital libraries: criteria and problems from users perspectives. Libr. Inf. Res. 28: 433–452 CrossRefGoogle Scholar
  71. 71.
    Klas, C., Albrechtsen, H.N.F., Hansen, P., Kapidakis, S., Kovacs, L., Kriewel, S., Micsik, A., Papatheodorou, C., Tsakonas, G., Jacob, E.: A logging scheme for comparative digital library evaluation. In: Lecture Notes In Computer Science, vol. 4172, pp. 267—278 Springer, Berlin (2006) Research and Advanced Technology for Digital Libraries: 10th European Conference, ECDL 2006, Alicante, Spain, September 17—22, 2006. ProceedingsGoogle Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  • Norbert Fuhr
    • 1
  • Giannis Tsakonas
    • 6
  • Trond Aalberg
    • 2
  • Maristella Agosti
    • 5
  • Preben Hansen
    • 3
  • Sarantos Kapidakis
    • 6
  • Claus-Peter Klas
    • 1
    Email author
  • László Kovács
    • 4
  • Monica Landoni
    • 8
  • András Micsik
    • 4
  • Christos Papatheodorou
    • 6
  • Carol Peters
    • 7
  • Ingeborg Sølvberg
    • 2
    • 6
  1. 1.University of Duisburg-EssenDuisburgGermany
  2. 2.Norwegian University of Science and TechnologyTrondheimNorway
  3. 3.Swedish Institute of Computer ScienceKistaSweden
  4. 4.Computer and Automation Research Institute of the Hungarian Academy of Sciences (MTA SZTAKI)BudapestHungary
  5. 5.University of PaduaPadovaItaly
  6. 6.Department of Archive and Library SciencesIonian UniversityCorfuGreece
  7. 7.CNR-ISTIPisaItaly
  8. 8.Department of Computer and Information SciencesUniversity of StrathclydeGlasgowUK

Personalised recommendations