Evaluating a digital humanities research environment: the CULTURA approach

Abstract

Digital humanities initiatives play an important role in making cultural heritage collections accessible to the global community of researchers and general public for the first time. Further work is needed to provide useful and usable tools to support users in working with those digital contents in virtual environments. The CULTURA project has developed a corpus agnostic research environment integrating innovative services that guide, assist and empower a broad spectrum of users in their interaction with cultural artefacts. This article presents (1) the CULTURA system and services and the two collections that have been used for testing and deploying the digital humanities research environment, and (2) an evaluation methodology and formative evaluation study with apprentice researchers. An evaluation model was developed which has served as a common ground for systematic evaluations of the CULTURA environment with user communities around the two test bed collections. The evaluation method has proven to be suitable for accommodating different evaluation strategies and allows meaningful consolidation of evaluation results. The evaluation outcomes indicate a positive perception of CULTURA. A range of useful suggestions for future improvement has been collected and fed back into the development of the next release of the research environment.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. 1.

    http://www.cultura-strep.eu/outcomes#2

  2. 2.

    http://www.europeana.eu/.

  3. 3.

    http://www.oldbaileyonline.org/.

  4. 4.

    http://criminalintent.org/.

  5. 5.

    http://voyant-tools.org/.

  6. 6.

    https://www.zotero.org/.

  7. 7.

    http://www.e-enlightenment.com/.

  8. 8.

    http://www.artstor.org/index.shtml.

  9. 9.

    http://pro.europeana.eu/web/europeana-cloud.

  10. 10.

    http://cultura-project.eu/.

  11. 11.

    http://www.drupal.org/.

  12. 12.

    http://1641.tcd.ie/.

  13. 13.

    http://www.mysql.com/.

  14. 14.

    http://ipsa.dei.unipd.it/en_GB/home.

  15. 15.

    http://cultura-project.eu/1916/.

References

  1. 1.

    Borgman, C.L.: The digital future is now: A call to action for the humanities. Digital Human. Quart. 3(4). http://www.digitalhumanities.org/dhq/vol/3/4/000077/000077.html (2009). Accessed 3 Oct 2013

  2. 2.

    Hampson, C., Agosti, M., Orio, N., Bailey, E., Lawless, S., Conlan, O., Wade, V.: The CULTURA project: supporting next generation interaction with digital cultural heritage collections. In: Proceedings of the 4th International Euromed Conference, pp. 668–675. Springer, Heidelberg (2012)

  3. 3.

    Hampson, C., Lawless, S., Bailey, E., Yogev, S., Zwerdling, N., Carmel, D.: CULTURA: A metadata-rich environment to support the enhanced interrogation of cultural collections. In: Dodero, J.M., Palomo-Duarte, M., Karmpiperis, P. (eds.) Proceedings of the 6th Metadata and Semantics Research Conference, pp. 227–238. Springer, Berlin (2012)

  4. 4.

    Clarke, A.: The 1641 Depositions. In: Fox, P. (ed.) Treasures of the Library, pp. 111–122. Trinity College, Dublin (1986)

    Google Scholar 

  5. 5.

    O’Regan, D., Sweetnam, M., Fennell, B., Lawless, S.: A collaborative linguistic research interface for the 1641 Depositions. Poster presented at Digital Humanities 2011. Stanford, CA (2011)

  6. 6.

    Agosti, M., Benfante, L., Orio, N.: IPSA: A digital archive of herbals to support scientific research. In: Sembok T.M.T. (ed.) Proceedings of the International Conference on Asian Digital Libraries ICADL 2003. LNCS vol. 2911, pp. 253–264. Springer, Berlin (2003)

  7. 7.

    Bellamy, C.: The sound of many hands clapping: Teaching the digital humanities through virtual research environments (VREs). Digital Human. Quart. 6(2). http://www.digitalhumanities.org/dhq/vol/6/2/000119/000119.html (2012). Accessed 3 Oct 2013

  8. 8.

    Hunter, J., Gerber, A.: The Aus-e-Lit project: Advanced eResearch services for scholars of Australian literature. VALA 2010. Melbourne, Australia. http://www.itee.uq.edu.au/eresearch/papers/2010/VALA2010_Hunter.pdf (2010). Accessed 3 Oct 2013

  9. 9.

    Neuroth, H., Lohmeier, F., Smith, K.M.: TextGrid—virtual research environment for the humanities. Int. J. Digit. Curation 6(2), 222–231 (2011)

    Article  Google Scholar 

  10. 10.

    Agosti, M., Conlan, O., Ferro, N., Hampson, C., Munnelly, G.: Interacting with digital cultural heritage collections via annotations: the CULTURA approach. In: Proceedings of the 2013 ACM Symposium on Document Engineering, pp. 13–22. ACM, New York (2013)

  11. 11.

    Lawless, S., Hampson, C., Mitankin, P., Gerdjikov, S.: Normalisation in historical text collections. In: Proceedings of Digital Humanities 2013, pp. 507–509. Lincoln, Nebraska (2013)

  12. 12.

    Yogev, S., Roitman, H., Carmel, D. Zwerdling, N.: Towards expressive exploratory search over entity-relationship data. In: Proceedings of the \(21^{st}\) International Conference Companion on World Wide Web, pp. 83–92. New York (2012)

  13. 13.

    Hampson, C., Bailey, E., Munnelly, G., Lawless, S., Conlan, O.: Dynamic personalisation for digital cultural heritage collections. In: Proceedings of the 6th International Workshop on Personalized Access to Cultural Heritage. Rome (2013)

  14. 14.

    Carmel, D., Zwerdling, N., Yogev, S.: Entity oriented search and exploration for cultural heritage collections: the EU CULTURA project. In: Proceedings of the \(21^{st}\) International Conference Companion on World Wide Web, pp. 227–230. New York (2012)

  15. 15.

    Mariani Canova, G.: Per Cultura: le immagini dei manoscritti della scienza a Padova dal Medioevo al Rinascimento. In: Atti e Memorie dell’Accademia Galileiana di Scienze, Lettere ed Arti. vol. CXXIV, pp. 81–90 (2011–2012)

  16. 16.

    Saracevic, T.: Digital library evaluation: toward an evolution of concepts. Libr. Trends 49(2), 350–369 (2000)

    Google Scholar 

  17. 17.

    Zhang, Y.: Developing a holistic model for digital library evaluation. Dissertation. The State University of New Jersey. (2007)

  18. 18.

    De Jong, M., Schellens, P.J.: Reader-focused text evaluation: an overview of goals and methods. J. Bus. Tech. Commun. 11(4), 401–432 (1997)

    Google Scholar 

  19. 19.

    Saracevic, T.: Evaluation of digital libraries: An overview. In: Agosti M., Fuhr N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries. Padua, Italy. http://www.scils.rutgers.edu/~tefko/DL_evaluation_Delos.pdf (2004). Accessed 7 Oct 2013

  20. 20.

    Kovács, L., Micsik, A.: The evaluation computer: A model for structuring evaluation activities. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries. Padua (2004)

  21. 21.

    Zhang, Y.: Developing a holistic model for digital library evaluation. J. Am. Soc. Inf. Sci. Technol. 61(1), 88–110 (2010)

    Article  Google Scholar 

  22. 22.

    Fuhr, N., Tsakonas, G., Aalberg, T., Agosti, M., Hansen, P., Kapidakis, S., Klas, C.-P., Kovács, L., Landoni, M., Micsik, A., Papatheodorou, C., Peters, C., Sølvberg, I.: Evaluation of digital libraries. Int. J. Digit. Libr. 8(1), 21–38 (2007)

    Article  Google Scholar 

  23. 23.

    Tsakonas, G., Papatheodorou, C.: Analysing and evaluating usefulness and usability in electronic information services. J. Inf. Sci. 32(5), 400–419 (2006)

    Article  Google Scholar 

  24. 24.

    Tsakonas, G., Kapidakis, S., Papatheodorou, C.: Evaluation of user interaction in digital libraries. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on Evaluation of Digital Libraries, pp. 45–60. Padua (2004)

  25. 25.

    Steiner, C.M., Hillemann, E., Nussbaumer, A., Albert, D., Sweetnam, M., Hampson, C., Conlan, O.: The CULTURA evaluation model: An approach responding to the evaluation needs in an innovative research environment. In: S. Lawless, M. Agosti, P. Clough, O. Conlan (eds.) Proceedings of the First Workshop on Exploration, Navigation and Retrieval of Information in Cultural Heritage, SIGIR 2013, pp. 43–46. ACM, Dublin (2013)

  26. 26.

    Sweetnam, M., Agosti, M., Orio, N., Ponchia, C., Steiner, C., Hillemann, E.-C., Ó Siochrú, M., Lawless, S.: User needs for enhanced engagement with cultural heritage collections. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds.) Theory and Practice of Digital Libraries, TPDL 2012. LNCS vol. 7489, pp. 64–75. Springer, Berlin (2012)

  27. 27.

    Davis, F.D., Bagozzi, R.P., Warshaw, P.R.: User acceptance of computer technology: a comparison of two theoretical models. Manag. Sci. 35(8), 982–1003 (1989)

    Article  Google Scholar 

  28. 28.

    Gerdjikov, S., Mihov, S., Nenchev, V.: Extraction of spelling variations from language structure for noisy text correction. In: Proceedings of the 12th International Conference on Document Analysis and Recognition. IEEE, pp. 324–328 (2013)

  29. 29.

    Steiner, C.M., Albert, D.: Tailor-made or unfledged? Evaluating the quality of adaptive eLearning. In: Psaromiligkos, Spyridakos, A., Retalis, S. (eds.) Evaluation in e-learning., pp. 111–143. Nova Science, New York (2012)

  30. 30.

    Brusilovsky, P., Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive learning systems. Int. J. Contin. Eng. Educ. Life-Long Learn. 14, 402–421 (2004)

    Article  Google Scholar 

  31. 31.

    Brooke, J.: SUS: a “quick and dirty” usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McCleeland, A.L. (eds.) Usability Evaluation in Industry, pp. 189–194. Taylor and Francis, London (1996)

    Google Scholar 

  32. 32.

    Thong, J.Y.L., Hong, W., Tam, K.-Y.: Understanding user acceptance of digital libraries: what are the roles of interface characteristics, organizational context, and individual differences? Int. J. Hum.-Comput. Stud. 57, 215–242 (2002)

    Article  Google Scholar 

  33. 33.

    Hampson, C., Lawless, S., Bailey, E., Steiner, C.M., Hillemann, E.-C., Conlan, O.: Metadata-enhanced exploration of digital cultural collections. Int. J. Metadata Semant. Ontol. 9, 155–167 (2014)

    Article  Google Scholar 

  34. 34.

    Nussbaumer, A., Hillemann, E.-C., Steiner, C.M., Albert, D.: An evaluation system for digital libraries. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds.) Theory and practice of digital libraries. Second International Conference, TPDL 2012. LNCS, vol. 7489, pp. 414–419. Springer, Berlin (2012)

Download references

Acknowledgments

The work reported has been partially supported by the CULTURA project, as part of the Seventh Framework Programme of the European Commission, Area “Digital Libraries and Digital Preservation” (ICT-2009.4.1), grant agreement no. 269973, and would not have been possible without the close collaboration between all CULTURA partners. http://www.cultura-strep.eu/partners.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Christina M. Steiner.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Steiner, C.M., Agosti, M., Sweetnam, M.S. et al. Evaluating a digital humanities research environment: the CULTURA approach. Int J Digit Libr 15, 53–70 (2014). https://doi.org/10.1007/s00799-014-0127-x

Download citation

Keywords

  • Virtual research environment
  • Digital humanities
  • Cultural heritage
  • Evaluation