Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns

  • Marco Dussin
  • Nicola Ferro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5714)


This paper discusses the evolution of large-scale evaluation campaigns and the corresponding evaluation infrastructures needed to carry them out. We present the next challenges for these initiatives and show how digital library systems can play a relevant role in supporting the research conducted in these fora by acting as virtual research environments.


Digital Library Information Resource Evaluation Campaign Relational Connection Knowledge Creation Process 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ackoff, R.L.: From Data to Wisdom. Journal of Applied Systems Analysis 16, 3–9 (1989)Google Scholar
  2. 2.
    Agosti, M., Di Nunzio, G.M., Ferro, N.: A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns. In: Proc. 1st International Workshop on Evaluating Information Access (EVIA 2007), pp. 62–73. National Institute of Informatics, Tokyo (2007)Google Scholar
  3. 3.
    Agosti, M., Di Nunzio, G.M., Ferro, N., Harman, D., Peters, C.: The Future of Large-scale Evaluation Campaigns for Information Retrieval in Europe. In: Kovács, L., Fuhr, N., Meghini, C. (eds.) ECDL 2007. LNCS, vol. 4675, pp. 509–512. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Agosti, M., Ferro, N.: A Formal Model of Annotations of Digital Content. ACM Transactions on Information Systems (TOIS) 26(1), 3:1–3:57 (2008)Google Scholar
  5. 5.
    Agosti, M., Ferro, N.: Towards an Evaluation Infrastructure for DL Performance Evaluation. In: Evaluation of Digital Libraries: An Insight to Useful Applications and Methods. Chandos Publishing (2009)Google Scholar
  6. 6.
    Cleverdon, C.W.: The Cranfield Tests on Index Languages Devices. In: Readings in Information Retrieval, pp. 47–60. Morgan Kaufmann Publisher, San Francisco (1997)Google Scholar
  7. 7.
    Di Nunzio, G.M., Ferro, N.: DIRECT: a System for Evaluating Information Access Components of Digital Libraries. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds.) ECDL 2005. LNCS, vol. 3652, pp. 483–484. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    Dussin, M., Ferro, N.: Design of a Digital Library System for Large-Scale Evaluation Campaigns. In: Christensen-Dalsgaard, B., Castelli, D., Ammitzbøll Jurik, B., Lippincott, J. (eds.) ECDL 2008. LNCS, vol. 5173, pp. 400–401. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Dussin, M., Ferro, N.: The Role of the DIKW Hierarchy in the Design of a Digital Library System for the Scientific Data of Large-Scale Evaluation Campaigns. In: Proc. 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), p. 450. ACM Press, New York (2008)Google Scholar
  10. 10.
    European Commission Information Society and Media. i2010: Digital Libraries (October 2006),
  11. 11.
    Ferro, N.: Digital Annotations: a Formal Model and its Applications. In: Information Access through Search Engines and Digital Libraries, pp. 113–146. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Ferro, N., Peters, C.: From CLEF to TrebleCLEF: the Evolution of the Cross-Language Evaluation Forum. In: Proc. 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access, pp. 577–593. National Institute of Informatics, Tokyo (2008)Google Scholar
  13. 13.
    Fricke, M.: The Knowledge Pyramid: a Critique of the DIKW Hierarchy. Journal of Information Science 35(2), 131–142 (2009)CrossRefGoogle Scholar
  14. 14.
    Fuhr, N., et al.: Evaluation of Digital Libraries. International Journal on Digital Libraries, 8(1):21–38 (2007)Google Scholar
  15. 15.
    Harman, D.K., Voorhess, E.M. (eds.): TREC. Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)Google Scholar
  16. 16.
    National Science Board. Long-Lived Digital Data Collections: Enabling Research and Education in the 21st Century (NSB-05-40). National Science Foundation (NSF) (September 2005),
  17. 17.
    Nonaka, I., Takeuchi, H.: The knowledge-creating company: How Japanese companies create the dynamics of innovation. Oxford University Press, USA (1995)Google Scholar
  18. 18.
    Rowley, J.: The Wisdom Hierarchy: Representations of the DIKW Hierarchy. Journal of Information Science 33(2), 163–180 (2007)CrossRefGoogle Scholar
  19. 19.
    Working Group on Data for Science. FROM DATA TO WISDOM: Pathways to Successful Data Management for Australian Science. Report to the Primw Minister’s Science, Engineering and Innovation Council (PMSEIC) (September 2006),
  20. 20.
    Zeleny, M.: Management Support Systems: Towards Integrated Knowledge Management. Human Systems Management 7(1), 59–70 (1987)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Marco Dussin
    • 1
  • Nicola Ferro
    • 1
  1. 1.University of PaduaItaly

Personalised recommendations