Assessing the Scholarly Impact of ImageCLEF

  • Theodora Tsikrika
  • Alba G. Seco de Herrera
  • Henning Müller
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6941)


Systematic evaluation has an important place in information retrieval research starting with the Cranfield tests and currently with TREC (Text REtrieval Conference) and other evaluation campaigns. Such benchmarks are often mentioned to have an important impact in advancing a research field and making techniques comparable. Still, their exact impact is hard to measure. This paper aims at assessing the scholarly impact of the ImageCLEF image retrieval evaluation initiative. To this end, the papers in the proceedings published after each evaluation campaign and their citations are analysed using Scopus and Google Scholar. A significant impact of ImageCLEF could be shown through this bibliometric analysis. The differences between the employed analysis methods, each with its advantages and limitations, are also discussed.


Information Retrieval Image Retrieval Citation Analysis Citation Count Bibliometric Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bar-Ilan, J.: Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics 74(2), 257–271 (2008)CrossRefGoogle Scholar
  2. 2.
    Cleverdon, C.W.: The evaluation of systems used in information retrieval. In: Proceedings of the International Conference on Scientific Information, vol. 1, pp. 687–698. National Academy of Sciences, National Research Council (1959)Google Scholar
  3. 3.
    Clough, P., Müller, H., Deselaers, T., Grubinger, M., Lehmann, T.M., Jensen, J., Hersh, W.R.: The CLEF 2005 cross–language image retrieval track. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 535–557. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    A.-W. Harzing. Citation analysis across disciplines: The impact of different data sources and citation metrics (2010), (retrieved)
  5. 5.
    Hirsch, J.E.: An index to quantify an individuals scientific research output. Proceedings of the National Academy of Sciences (PNAS) 102(46), 16569–16572 (2005)CrossRefGoogle Scholar
  6. 6.
    Proceedings of the 20th International Conference on Pattern Recognition (ICPR 2010). IEEE Computer Society, Instanbul (2010)Google Scholar
  7. 7.
    Jacsó, P.: Deflated, inflated and phantom citation counts. Online Information Review 30(3), 297–309 (2006)CrossRefGoogle Scholar
  8. 8.
    Jacsó, P.: The pros and cons of computing the h-index using Google Scholar. Online Information Review 32(3), 437–452 (2008)CrossRefGoogle Scholar
  9. 9.
    Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF: Experimental Evaluation in Visual Information Retrieval, 1st edn. Springer, Heidelberg (2010)zbMATHGoogle Scholar
  10. 10.
    Rahm, E., Thor, A.: Citation analysis of database publications. SIGMOD Record 34, 48–53 (2005)CrossRefGoogle Scholar
  11. 11.
    Rowe, B.R., Wood, D.W., Link, A.N., Simoni, D.A.: Economic impact assessment of NIST’s Text REtrieval Conference (TREC) Program. Technical Report Project Number 0211875, RTI International (2010)Google Scholar
  12. 12.
    Sanderson, M.: Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval 4, 247–375 (2010)CrossRefzbMATHGoogle Scholar
  13. 13.
    Thornley, C.V., Johnson, A.C., Smeaton, A.F., Lee, H.: The scholarly impact of TRECVid (2003–2009). JASIST 62(4), 613–627 (2011)CrossRefGoogle Scholar
  14. 14.
    Ünay, D., Çataltepe, Z., Aksoy, S. (eds.): Proceedings of the 20th International Conference on Recognizing Patterns in Signals, Speech, Images, and Videos, ICPR Contest Reports. Springer, Heidelberg (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Theodora Tsikrika
    • 1
  • Alba G. Seco de Herrera
    • 1
  • Henning Müller
    • 1
  1. 1.University of Applied Sciences Western Switzerland (HES–SO)SierreSwitzerland

Personalised recommendations