The Scholarly Impact of CLEF 2010–2017

A Google Scholar Analysis of CLEF Proceedings and Working Notes
  • Birger LarsenEmail author
Part of the The Information Retrieval Series book series (INRE, volume 41)


This chapter assesses the scholarly impact of the CLEF evaluation campaign by performing a bibliometric analysis of the citations of the CLEF 2010–2017 papers collected through Google Scholar. The analysis extends an earlier 2013 study by Tsikrika et al. of the CLEF Proceedings for the period 2000–2009 and compares the impact of the first half of CLEF to the second. It also extends the analysis by including the CLEF Working notes, a less formal but important part of the CLEF oeuvre. Results show that, despite the different nature of the peer-reviewed CLEF Proceedings papers and the less formal and much more numerous Working note papers, both types of publications have high citation impact. In particular, overview papers from the various labs and tasks in CLEF attract large amounts of citations in both Proceedings and Working Notes. A significant proportion of the total number of citations appear to be from outside CLEF—there are simply not enough CLEF papers every year to explain that many citations. In conclusion, the analysis of the productivity and citation impact of CLEF in the period 2010–2017 shows that CLEF is a very strong and vibrant initiative that has managed a major change of format between 2009/2010 and continues to produce relevant research, datasets and tools.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



We wish to thank Lucas Chaves, University of Copenhagen, Denmark for assistance in matching DBLP and Google Scholar data, Anne-Wil Harzing for help with the ‘Publish or Perish’ software platform, as well as two anonymous reviewers for constructive feedback.


  1. Tsikrika T, Larsen B, Müller H, Endrullis S, Rahm E (2013) The scholarly impact of CLEF (2000–2009). In: Forner P, Müller H, Paredes R, Rosso P, Stein B (eds) Information access evaluation meets multilinguality, multimodality, and visualization. Proceedings of the fourth international conference of the CLEF initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138. Springer, Heidelberg, pp 1–12CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Science, Policy and Information Studies, Department of Communication and PsychologyAalborg UniversityCopenhagenDenmark

Personalised recommendations