A Logging Scheme for Comparative Digital Library Evaluation

  • Claus-Peter Klas
  • Hanne Albrechtsen
  • Norbert Fuhr
  • Preben Hansen
  • Sarantos Kapidakis
  • Laszlo Kovacs
  • Sascha Kriewel
  • Andras Micsik
  • Christos Papatheodorou
  • Giannis Tsakonas
  • Elin Jacob
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4172)

Abstract

Evaluation of digital libraries assesses their effectiveness, quality and overall impact. To facilitate the comparison of different evaluations and to support the re-use of evaluation data, we are proposing a new logging schema. This schema will allow for logging and sharing of a wide array of data about users, systems and their interactions. We discuss the multi-level logging framework presented in [19] and describe how the community can add to and gain from using the framework. The main focus of this paper is the logging of events within digital libraries on a generalised, conceptual level, as well as the services based on it. These services will allow diverse digital libraries to store their log data in a common repository using a common format. In addition they provide means for analysis and comparison of search history data.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bates, M.J.: Idea tactics. JASIS 30(5), 280–289 (1979)CrossRefGoogle Scholar
  2. 2.
    Bates, M.J.: Information search tactics. JASIS 30(4), 205–214 (1979)CrossRefGoogle Scholar
  3. 3.
    Bertini, E., Catarci, T., Kimani, S., Bello, L.D.: Visualization in digital libraries. From Integrated Publication and Information Systems to Virtual Information and Knowledge Environments, pp. 183–196 (2005)Google Scholar
  4. 4.
    Bertot, J., McClure, C., Moen, W., Rubin, J.: Web usage statistics: measurement issues and analytical techniques. Goverment Information Quarterly 14(4), 373–395 (1997)CrossRefGoogle Scholar
  5. 5.
    Cooper, M.D.: Design considerations in instrumenting and monitoring web-based information retrieval systems. JASIS 49(10), 903–919 (1998)CrossRefGoogle Scholar
  6. 6.
    Cooper, M.D.: Usage patterns of a web-based library catalog. JASIST 52(2), 137–148 (2001)CrossRefGoogle Scholar
  7. 7.
    Covey, D.T.: Usage and usability assessment: Library practices and concerns. Technical report, Digital Library Federation (2002)Google Scholar
  8. 8.
    Fuhr, N., Klas, C.-P., Schaefer, A., Mutschke, P.: Daffodil: An integrated desktop for supporting high-level search activities in federated digital libraries. In: Agosti, M., Thanos, C. (eds.) ECDL 2002. LNCS, vol. 2458, pp. 597–612. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Fuhr, N., Tsakonas, G., Aalberg, T., Agosti, M., Hansen, P., Kapidakis, S., Klas, C.-P., Kovcs, L., Landoni, M., Micsik, A., Papatheodorou, C., Peters, C., Solvberg, I.: Evaluation of digital libraries (2006)Google Scholar
  10. 10.
    Golder, S.A., Huberman, B.A.: The structure of collaborative tagging systems. Technical report, Information Dynamics Labs, HP Labs (2005)Google Scholar
  11. 11.
    Goncalves, M.A., Fox, E.A., Cassel, L., Krowne, A., Ravindranathan, U., Panchanathan, G., Jagodzinski, F.: Standards, mark-up, and metadata: The xml log standard for digital libraries: analysis, evolution, and deployment. In: Proceedings of the third ACM/IEEE-CS joint conference on Digital libraries (2003)Google Scholar
  12. 12.
    Gonçalves, M.A., Luo, M., Shen, R., Ali, M.F., Fox, E.A.: An XML log standard and tool for digital library logging analysis. In: Agosti, M., Thanos, C. (eds.) ECDL 2002. LNCS, vol. 2458, pp. 129–143. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Hearst, M.A., Ivory, M.Y.: The state of the art in automating usability evaluation of user interfaces, pp. 470–516 (2001)Google Scholar
  14. 14.
    Helms, J., Neale, D., Isenhour, P., Carroll, J.M.: Data logging: Higher-level capturing and multi-level abstracting of user activities. In: Proceedings of the 40th annual meeting of the Human Factors and Ergonomics Society (2000)Google Scholar
  15. 15.
    Hilbert, D.M., Redmiles, D.F.: Extracting usability information from user interface events. ACM Computing Surveys 32(4), 384–421 (2000)CrossRefGoogle Scholar
  16. 16.
    John, B.E., Kieras, D.E.: Using goms for user interface design and evaluation: Which technique? ACM Trans. Comput.-Hum. Interact. 3 (1996)Google Scholar
  17. 17.
    Jones, S., Cunningham, S.J., McNab, R., Boddie, S.: A transaction log analysis of a digital library. International Journal on Digital Libraries 3(2), 152–169 (2000)CrossRefGoogle Scholar
  18. 18.
    Ke, H., Kwakkelaar, R., Tai, T., Chen, L.: Exploring behavior of e-journal users in science and technology: Transaction log analysis of elsevier’s sciencedirect onsite in taiwan. Library & Information Science Research 24(3), 265–291 (2002)CrossRefGoogle Scholar
  19. 19.
    Klas, C.-P., Albrechtsen, H., Fuhr, N., Hansen, P., Jacob, E., Kapidakis, S., Kovacs, L., Kriewel, S., Micsik, A., Papatheodorou, C., Tsakonas, G.: An Experimental Framework for Comparative Digital Library Evaluation: The Logging Scheme. In: JCDL, Short Paper (2006)Google Scholar
  20. 20.
    Kriewel, S.: Finding and using strategies for search situations in digital libraries. Bulletin of the IEEE Technical Committee on Digital Libraries (2005) (to appear)Google Scholar
  21. 21.
    Kriewel, S., Klas, C.-P., Schaefer, A., Fuhr, N.: Daffodil - strategic support for user-oriented access to heterogeneous digital libraries. D-Lib Magazine 10(6) (June 2004)Google Scholar
  22. 22.
    Pan, B.: Capturing users behavior in the national science digital library (nsdl). Technical report, NSDL (2003)Google Scholar
  23. 23.
    Schaefer, A., Jordan, M., Klas, C.-P., Fuhr, N.: Active support for query formulation in virtual digital libraries: A case study with DAFFODIL. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds.) ECDL 2005. LNCS, vol. 3652, pp. 414–425. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  24. 24.
    Sfakakis, M., Kapidakis, S.: User Behavior Tendencies on Data Collections in a Digital Library. In: Agosti, M., Thanos, C. (eds.) ECDL 2002. LNCS, vol. 2458, pp. 550–559. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  25. 25.
    Shneiderman, B.: The eyes have it: A task by data type taxonomy for information visualizations. Technical Report CS-TR-3665, University of Maryland, Department of Computer Science (July 1996)Google Scholar
  26. 26.
    Wildemuth, B.M.: The effects of domain knowledge on search tactic formulation. JASIST 55(3), 246–258 (2004)CrossRefGoogle Scholar
  27. 27.
    Yuan, W., Meadow, C.T.: A study of the use of variables in information retrieval user studies. JASIS 50(2), 140–150 (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Claus-Peter Klas
    • 1
  • Hanne Albrechtsen
    • 2
  • Norbert Fuhr
    • 1
  • Preben Hansen
    • 5
  • Sarantos Kapidakis
    • 4
  • Laszlo Kovacs
    • 3
  • Sascha Kriewel
    • 1
  • Andras Micsik
    • 3
  • Christos Papatheodorou
    • 4
  • Giannis Tsakonas
    • 4
  • Elin Jacob
    • 6
  1. 1.University of Duisburg-EssenDuisburgGermany
  2. 2.Institute of Knowledge SharingCopenhagenDenmark
  3. 3.MTA SZTAKIBudapestHungary
  4. 4.Ionian UniversityKekryraGreece
  5. 5.Swedish Institute of Computer ScienceKistaSweden
  6. 6.Indiana University BloomingtonBloomingtonUSA

Personalised recommendations