Abstract
Systematic evaluation has an important place in information retrieval research starting with the Cranfield tests and currently with TREC (Text REtrieval Conference) and other evaluation campaigns. Such benchmarks are often mentioned to have an important impact in advancing a research field and making techniques comparable. Still, their exact impact is hard to measure. This paper aims at assessing the scholarly impact of the ImageCLEF image retrieval evaluation initiative. To this end, the papers in the proceedings published after each evaluation campaign and their citations are analysed using Scopus and Google Scholar. A significant impact of ImageCLEF could be shown through this bibliometric analysis. The differences between the employed analysis methods, each with its advantages and limitations, are also discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bar-Ilan, J.: Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics 74(2), 257–271 (2008)
Cleverdon, C.W.: The evaluation of systems used in information retrieval. In: Proceedings of the International Conference on Scientific Information, vol. 1, pp. 687–698. National Academy of Sciences, National Research Council (1959)
Clough, P., Müller, H., Deselaers, T., Grubinger, M., Lehmann, T.M., Jensen, J., Hersh, W.R.: The CLEF 2005 cross–language image retrieval track. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 535–557. Springer, Heidelberg (2006)
A.-W. Harzing. Citation analysis across disciplines: The impact of different data sources and citation metrics (2010), http://www.harzing.com/data_metrics_comparison.htm (retrieved)
Hirsch, J.E.: An index to quantify an individuals scientific research output. Proceedings of the National Academy of Sciences (PNAS) 102(46), 16569–16572 (2005)
Proceedings of the 20th International Conference on Pattern Recognition (ICPR 2010). IEEE Computer Society, Instanbul (2010)
Jacsó, P.: Deflated, inflated and phantom citation counts. Online Information Review 30(3), 297–309 (2006)
Jacsó, P.: The pros and cons of computing the h-index using Google Scholar. Online Information Review 32(3), 437–452 (2008)
Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF: Experimental Evaluation in Visual Information Retrieval, 1st edn. Springer, Heidelberg (2010)
Rahm, E., Thor, A.: Citation analysis of database publications. SIGMOD Record 34, 48–53 (2005)
Rowe, B.R., Wood, D.W., Link, A.N., Simoni, D.A.: Economic impact assessment of NIST’s Text REtrieval Conference (TREC) Program. Technical Report Project Number 0211875, RTI International (2010)
Sanderson, M.: Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval 4, 247–375 (2010)
Thornley, C.V., Johnson, A.C., Smeaton, A.F., Lee, H.: The scholarly impact of TRECVid (2003–2009). JASIST 62(4), 613–627 (2011)
Ünay, D., Çataltepe, Z., Aksoy, S. (eds.): Proceedings of the 20th International Conference on Recognizing Patterns in Signals, Speech, Images, and Videos, ICPR Contest Reports. Springer, Heidelberg (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tsikrika, T., de Herrera, A.G.S., Müller, H. (2011). Assessing the Scholarly Impact of ImageCLEF. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2011. Lecture Notes in Computer Science, vol 6941. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23708-9_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-23708-9_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23707-2
Online ISBN: 978-3-642-23708-9
eBook Packages: Computer ScienceComputer Science (R0)