A Logging Scheme for Comparative Digital Library Evaluation
Evaluation of digital libraries assesses their effectiveness, quality and overall impact. To facilitate the comparison of different evaluations and to support the re-use of evaluation data, we are proposing a new logging schema. This schema will allow for logging and sharing of a wide array of data about users, systems and their interactions. We discuss the multi-level logging framework presented in  and describe how the community can add to and gain from using the framework. The main focus of this paper is the logging of events within digital libraries on a generalised, conceptual level, as well as the services based on it. These services will allow diverse digital libraries to store their log data in a common repository using a common format. In addition they provide means for analysis and comparison of search history data.
Unable to display preview. Download preview PDF.
- 3.Bertini, E., Catarci, T., Kimani, S., Bello, L.D.: Visualization in digital libraries. From Integrated Publication and Information Systems to Virtual Information and Knowledge Environments, pp. 183–196 (2005)Google Scholar
- 7.Covey, D.T.: Usage and usability assessment: Library practices and concerns. Technical report, Digital Library Federation (2002)Google Scholar
- 9.Fuhr, N., Tsakonas, G., Aalberg, T., Agosti, M., Hansen, P., Kapidakis, S., Klas, C.-P., Kovcs, L., Landoni, M., Micsik, A., Papatheodorou, C., Peters, C., Solvberg, I.: Evaluation of digital libraries (2006)Google Scholar
- 10.Golder, S.A., Huberman, B.A.: The structure of collaborative tagging systems. Technical report, Information Dynamics Labs, HP Labs (2005)Google Scholar
- 11.Goncalves, M.A., Fox, E.A., Cassel, L., Krowne, A., Ravindranathan, U., Panchanathan, G., Jagodzinski, F.: Standards, mark-up, and metadata: The xml log standard for digital libraries: analysis, evolution, and deployment. In: Proceedings of the third ACM/IEEE-CS joint conference on Digital libraries (2003)Google Scholar
- 13.Hearst, M.A., Ivory, M.Y.: The state of the art in automating usability evaluation of user interfaces, pp. 470–516 (2001)Google Scholar
- 14.Helms, J., Neale, D., Isenhour, P., Carroll, J.M.: Data logging: Higher-level capturing and multi-level abstracting of user activities. In: Proceedings of the 40th annual meeting of the Human Factors and Ergonomics Society (2000)Google Scholar
- 16.John, B.E., Kieras, D.E.: Using goms for user interface design and evaluation: Which technique? ACM Trans. Comput.-Hum. Interact. 3 (1996)Google Scholar
- 19.Klas, C.-P., Albrechtsen, H., Fuhr, N., Hansen, P., Jacob, E., Kapidakis, S., Kovacs, L., Kriewel, S., Micsik, A., Papatheodorou, C., Tsakonas, G.: An Experimental Framework for Comparative Digital Library Evaluation: The Logging Scheme. In: JCDL, Short Paper (2006)Google Scholar
- 20.Kriewel, S.: Finding and using strategies for search situations in digital libraries. Bulletin of the IEEE Technical Committee on Digital Libraries (2005) (to appear)Google Scholar
- 21.Kriewel, S., Klas, C.-P., Schaefer, A., Fuhr, N.: Daffodil - strategic support for user-oriented access to heterogeneous digital libraries. D-Lib Magazine 10(6) (June 2004)Google Scholar
- 22.Pan, B.: Capturing users behavior in the national science digital library (nsdl). Technical report, NSDL (2003)Google Scholar
- 25.Shneiderman, B.: The eyes have it: A task by data type taxonomy for information visualizations. Technical Report CS-TR-3665, University of Maryland, Department of Computer Science (July 1996)Google Scholar