Skip to main content
SpringerLink
Log in

What Happened in CLEF\(\ldots \) For a While?

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11696))

Abstract

2019 marks the 20\(^\text {th}\) birthday for CLEF, an evaluation campaign activity which has applied the Cranfield evaluation paradigm to the testing of multilingual and multimodal information access systems in Europe. This paper provides a summary of the motivations which led to the establishment of CLEF, and a description of how it has evolved over the years, the major achievements, and what we see as the next challenges.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.clef-initiative.eu/.

  2. 2.

    http://trec.nist.gov/.

  3. 3.

    http://research.nii.ac.jp/ntcir/.

  4. 4.

    http://fire.irsi.res.in/.

  5. 5.

    https://link.springer.com/conference/clef.

  6. 6.

    http://ceur-ws.org/.

  7. 7.

    http://www.multimediaeval.org/.

  8. 8.

    https://inex.mmci.uni-saarland.de/.

  9. 9.

    http://www.clef-initiative.eu/association.

  10. 10.

    http://www.promise-noe.eu/.

References

  1. Abnar, S., Dehghani, M., Shakery, A.: Meta text aligner: text alignment based on predicted plagiarism relation. In: Mothe et al. [301], pp. 193–199

    Google Scholar 

  2. Adams, J., Bedrick, S.: Automatic indexing of journal abstracts with latent semantic analysis. In: Mothe et al. [301], pp. 200–208

    Google Scholar 

  3. Afzal, Z., Akhondi, S.A., van Haagen, H.H.H.B.M., van Mulligen, E.M., Kors, J.A.: Concept recognition in French biomedical text using automatic translation. In: Fuhr et al. [143], pp. 162–173

    Google Scholar 

  4. Agirre, E., Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2008: ad hoc track overview. In: Peters et al. [359], pp. 15–37

    Google Scholar 

  5. Agirre, E., Di Nunzio, G.M., Mandl, T., Otegi, A.: CLEF 2009 ad hoc track overview: robust-WSD task. In: Peters et al. [360], pp. 36–49

    Google Scholar 

  6. Agirre, E., de Lacalle, O.L., Magnini, B., Otegi, A., Rigau, G., Vossen, P.: SemEval-2007 task 01: evaluating WSD on cross-language information retrieval. In: Peters et al. [366], pp. 908–917

    Google Scholar 

  7. Agosti, M., Di Buccio, E., Ferro, N., Masiero, I., Peruzzo, S., Silvello, G.: DIRECTions: design and specification of an IR evaluation infrastructure. In: Catarci et al. [75], pp. 88–99

    Google Scholar 

  8. Agosti, M., Di Nunzio, G.M., Ferro, N., Silvello, G.: An innovative approach to data management and curation of experimental data generated through IR test collections. In: Ferro and Peters [130]

    Google Scholar 

  9. Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A. (eds.): CLEF 2010. LNCS, vol. 6360. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15998-5

    Book  Google Scholar 

  10. Carrillo-de Albornoz, J., Gonzalo, J., Amigó, E.: RepLab: an evaluation campaign for online monitoring systems. In: Ferro and Peters [130]

    Google Scholar 

  11. Alfalahi, A., Eriksson, G., Sneiders, E.: Shadow answers as an intermediary in email answer retrieval. In: Mothe et al. [301], pp. 209–214

    Google Scholar 

  12. Alharbi, A., Stevenson, M.: Improving ranking for systematic reviews using query adaptation. In: Crestani et al. [94]

    Google Scholar 

  13. Alkhawaldeh, R.S., Jose, J.M.: Experimental study on semi-structured peer-to-peer information retrieval network. In: Mothe et al. [301], pp. 3–14

    Google Scholar 

  14. Allan, J., et al.: Research frontiers in information retrieval - report from the third strategic workshop on information retrieval in Lorne (SWIRL 2018). In: SIGIR Forum, vol. 52, no. 1, pp. 34–90 (2018)

    Google Scholar 

  15. Amigó, E., et al.: Overview of RepLab 2013: evaluating online reputation monitoring systems. In: Forner et al. [138], pp. 333–352

    Google Scholar 

  16. Amigó, E., et al.: Overview of RepLab 2014: author profiling and reputation dimensions for online reputation management. In: Kanoulas et al. [216], pp. 307–322

    Google Scholar 

  17. Amigó, E., Artiles, J., Gonzalo, J., Spina, D., Liu, B., Corujo, A.: WePS3 evaluation campaign: overview of the on-line reputation management task. In: Braschler et al. [61]

    Google Scholar 

  18. Amigó, E., Corujo, A., Gonzalo, J., Meij, E., de Rijke, M.: Overview of RepLab 2012: evaluating online reputation management systems. In: Forner et al. [137]

    Google Scholar 

  19. Amigó, E., Gonzalo, J., Verdejo, M.F.: A comparison of evaluation metrics for document filtering. In: Forner et al. [136], pp. 38–49

    Google Scholar 

  20. Anderka, M., Stein, B.: Overview of the 1st international competition on quality flaw prediction in Wikipedia. In: Forner et al. [137]

    Google Scholar 

  21. Andrearczyk, V., Müller, H.: Deep multimodal classification of image types in biomedical journal figures. In: Bellot et al. [42], pp. 3–14

    Google Scholar 

  22. Angelini, M., et al.: Cumulated relative position: a metric for ranking evaluation. In: Catarci et al. [75], pp. 112–123

    Google Scholar 

  23. Angelini, M., et al.: Measuring and analyzing the scholarly impact of experimental evaluation initiatives. In: Agosti, M., Catarci, T., Esposito, F. (eds.) Proceedings of the 10th Italian Research Conference on Digital Libraries (IRCDL 2014), vol. 38, pp. 133–137. Procedia Computer Science (2014)

    Google Scholar 

  24. Angelini, M., Ferro, N., Santucci, G., Silvello, G.: Improving ranking evaluation employing visual analytics. In: Forner et al. [138], pp. 29–40

    Google Scholar 

  25. Antunes, H., Lopes, C.: Analyzing the adequacy of readability indicators to a non-English language. In: Crestani et al. [94]

    Google Scholar 

  26. Argamon, S., Juola, P.: Overview of the international authorship identification competition at PAN-2011. In: Petras et al. [373]

    Google Scholar 

  27. Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Cheung, D.W.L., Song, I.Y., Chu, W.W., Hu, X., Lin, J.J. (eds.) Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM 2009), pp. 601–610. ACM Press, New York (2009)

    Google Scholar 

  28. Arni, T., Clough, P., Sanderson, M., Grubinger, M.: Overview of the ImageCLEFphoto 2008 photographic retrieval task. In: Peters et al. [359], pp. 500–511

    Google Scholar 

  29. Arora, P., Foster, J., Jones, G.J.F.: Query expansion for sentence retrieval using pseudo relevance feedback and word embedding. In: Jones et al. [209], pp. 97–103

    Google Scholar 

  30. Artiles, J., Borthwick, A., Gonzalo, J., Sekine, S., Amigó, E.: WePS-3 evaluation campaign: overview of the web people search clustering and attribute extraction tasks. In: Braschler et al. [61]

    Google Scholar 

  31. Azarbonyad, H., Marx, M.: How many labels? Determining the number of labels in multi-label text classification. In: Crestani et al. [94]

    Google Scholar 

  32. Azarbonyad, H., Saan, F., Dehghani, D., Marx, M., Kamps, J.: Are topically diverse documents also interesting? In: Mothe et al. [301], pp. 215–221

    Google Scholar 

  33. Azarbonyad, H., Shakery, A., Faili, H.: Exploiting multiple translation resources for English-Persian cross language information retrieval. In: Forner et al. [138], pp. 93–99

    Google Scholar 

  34. Azzopardi, L., Balog, K.: Towards a living lab for information retrieval research and development - a proposal for a living lab for product search tasks. In: Forner et al. [136], pp. 26–37

    Google Scholar 

  35. Balog, K., Azzopardi, L., Kamps, J., de Rijke, M.: Overview of WebCLEF 2006. In: Peters et al. [357], pp. 803–819

    Google Scholar 

  36. Balog, K., Cappellato, L., Ferro, N., Macdonald, C. (eds.): CLEF 2016 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2016). ISSN 1613-0073. http://ceur-ws.org/Vol-1609/

  37. Baradaran Hashemi, H., Shakery, A., Feili, H.: Creating a Persian-English comparable corpus. In: Agosti et al. [9], pp. 27–39

    Google Scholar 

  38. Basile, A., Dwyer, G., Medvedeva, M., Rawee, J., Haagsma, H., Nissim, M.: Simply the best: minimalist system trumps complex models in author profiling. In: Bellot et al. [42], pp. 143–156

    Google Scholar 

  39. Baudis, P., Sedivý, J.: Modeling of the question answering task in the YodaQA system. In: Mothe et al. [301], pp. 222–228

    Google Scholar 

  40. Bellot, P., et al.: Overview of INEX 2014. In: Kanoulas et al. [216], pp. 212–228

    Google Scholar 

  41. Bellot, P., et al.: Overview of INEX 2013. In: Forner et al. [138], pp. 269–281

    Google Scholar 

  42. Bellot, P., et al. (eds.): CLEF 2018. LNCS, vol. 11018. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7

    Book  Google Scholar 

  43. Beloborodov, A., Braslavski, P., Driker, M.: Towards automatic evaluation of health-related CQA data. In: Kanoulas et al. [216], pp. 7–18

    Google Scholar 

  44. Bensalem, I., Rosso, P., Chikhi, S.: A new corpus for the evaluation of Arabic intrinsic plagiarism detection. In: Forner et al. [138], pp. 53–58

    Google Scholar 

  45. Berendsen, R., Tsagkias, M., de Rijke, M., Meij, E.: Generating pseudo test collections for learning to rank scientific articles. In: Catarci et al. [75], pp. 42–53

    Google Scholar 

  46. Berlanga Llavori, R., Jimeno-Yepes, A., Pérez Catalán, M., Rebholz-Schuhmann, D.: Context-dependent semantic annotation in cross-lingual biomedical resources. In: Forner et al. [138], pp. 120–123

    Google Scholar 

  47. Berlanga Llavori, R., Pérez Catalán, M., Museros Cabedo, L., Forcada, R.: Semantic discovery of resources in cloud-based PACS/RIS systems. In: Forner et al. [138], pp. 167–178

    Google Scholar 

  48. Besançon, R., Chaudiron, S., Mostefa, D., Hamon, O., Timimi, I., Choukri, K.: Overview of CLEF 2008 INFILE pilot track. In: Peters et al. [359], pp. 939–946

    Google Scholar 

  49. Besançon, R., Chaudiron, S., Mostefa, D., Timimi, I., Choukri, K., Laïb, M.: Information filtering evaluation: overview of CLEF 2009 INFILE track. In: Peters et al. [360], pp. 342–353

    Google Scholar 

  50. Bhaskar, P., Bandyopadhyay, S.: Language independent query focused snippet generation. In: Catarci et al. [75], pp. 138–140

    Chapter  Google Scholar 

  51. Blinov, V., Mishchenko, K., Bolotova, V., Braslavski, P.: A pinch of humor for short-text conversation: an information retrieval approach. In: Jones et al. [209], pp. 3–15

    Google Scholar 

  52. Bordea, G., Thiessard, F., Hamon, T., Mougin, F.: Automatic query selection for acquisition and discovery of food-drug interactions. In: Bellot et al. [42], pp. 115–120

    Google Scholar 

  53. Borri, F., Nardi, A., Peters, C., Ferro, N. (eds.): CLEF 2008 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2008). ISSN 1613-0073. http://ceur-ws.org/Vol-1174/

  54. Borri, F., Nardi, A., Peters, C., Ferro, N. (eds.): CLEF 2009 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2009). ISSN 1613-0073. http://ceur-ws.org/Vol-1175/

  55. Borri, F., Peters, C., Ferro, N. (eds.): CLEF 2004 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2004). ISSN 1613-0073. http://ceur-ws.org/Vol-1170/

  56. Braschler, M.: CLEF 2000 - overview of results. In: Peters [345], pp. 89–101

    Chapter  Google Scholar 

  57. Braschler, M.: CLEF 2001 - overview of results. In: Peters et al. [354], pp. 9–26

    Google Scholar 

  58. Braschler, M.: CLEF 2002 - overview of results. In: Peters et al. [355], pp. 9–27

    Chapter  Google Scholar 

  59. Braschler, M.: CLEF 2003 - overview of results. In: Peters et al. [356], pp. 44–63

    Google Scholar 

  60. Braschler, M., Di Nunzio, G.M., Ferro, N., Peters, C.: CLEF 2004: ad hoc track overview and results analysis. In: Peters et al. [358], pp. 10–26

    Google Scholar 

  61. Braschler, M., Harman, D.K., Pianta, E., Ferro, N. (eds.): CLEF 2010 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2010). ISSN 1613-0073. http://ceur-ws.org/Vol-1176/

  62. Braschler, M., Peters, C.: Cross-language evaluation forum: objectives, results, achievements. Inf. Retrieval 7(1–2), 7–31 (2004)

    Article  Google Scholar 

  63. Braslavski, P., Verberne, S., Talipov, R.: Show me how to tie a tie: evaluation of cross-lingual video retrieval. In: Fuhr et al. [143], pp. 3–15

    Google Scholar 

  64. Budíková, P., Batko, M., Botorek, J., Zezula, P.: Search-based image annotation: extracting semantics from similar images. In: Mothe et al. [301], pp. 327–339

    Google Scholar 

  65. Buraya, K., Farseev, A., Filchenkov, A.: Multi-view personality profiling based on longitudinal data. In: Bellot et al. [42], pp. 15–27

    Google Scholar 

  66. Cabanac, G., Hubert, G., Boughanem, M., Chrisment, C.: Tie-breaking bias: effect of an uncontrolled parameter on information retrieval evaluation. In: Agosti et al. [9], pp. 112–123

    Google Scholar 

  67. Cappellato, L., Ferro, N., Goeuriot, L., Mandl, T. (eds.): CLEF 2017 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2017). ISSN 1613-0073. http://ceur-ws.org/Vol-1866/

  68. Cappellato, L., Ferro, N., Halvey, M., Kraaij, W. (eds.): CLEF 2014 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2014). ISSN 1613-0073. http://ceur-ws.org/Vol-1180/

  69. Cappellato, L., Ferro, N., Jones, G.J.F., SanJuan, E. (eds.): CLEF 2015 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2015). ISSN 1613-0073. http://ceur-ws.org/Vol-1391/

  70. Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.): CLEF 2019 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2019). ISSN 1613-0073. http://ceur-ws.org/Vol-2380/

  71. Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.): CLEF 2018 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2018). ISSN 1613-0073. http://ceur-ws.org/Vol-2125/

  72. Caputo, B., et al.: ImageCLEF 2014: overview and analysis of the results. In: Kanoulas et al. [216], pp. 192–211

    Google Scholar 

  73. Caputo, B., et al.: ImageCLEF 2013: the vision, the data and the open challenges. In: Forner et al. [138], pp. 250–268

    Google Scholar 

  74. Cassidy, T., Ji, H., Deng, H., Zheng, J., Han, J.: Analysis and refinement of cross-lingual entity linking. In: Catarci et al. [75], pp. 1–12

    Google Scholar 

  75. Catarci, T., Forner, P., Hiemstra, D., Peñas, A., Santucci, G. (eds.): CLEF 2012. LNCS, vol. 7488. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33247-0

    Book  Google Scholar 

  76. Chaa, M., Nouali, O., Bellot, P.: Combining tags and reviews to improve social book search performance. In: Bellot et al. [42], pp. 64–75

    Google Scholar 

  77. Chappell, T., Geva, S.: Overview of the INEX 2012 relevance feedback track. In: Forner et al. [137]

    Google Scholar 

  78. Chidlovskii, B., Csurka, G., Clinchant, S.: Evaluating stacked marginalised denoising autoencoders within domain adaptation methods. In: Mothe et al. [301], pp. 15–27

    Google Scholar 

  79. Chikka, V.R., Mariyasagayam, N., Niwa, Y., Karlapalem, K.: Information extraction from clinical documents: towards disease/disorder template filling. In: Mothe et al. [301], pp. 389–401

    Google Scholar 

  80. Chuklin, A., Severyn, A., Trippas, J., Alfonseca, E., Silen, H., Spina, D.: Using audio transformations to improve comprehension in voice question answering. In: Crestani et al. [94]

    Google Scholar 

  81. Cimiano, P., Lopez, V., Unger, C., Cabrio, E., Ngonga Ngomo, A.C., Walter, S.: Multilingual question answering over linked data (QALD-3): lab overview. In: Forner et al. [138], pp. 321–332

    Google Scholar 

  82. Clough, P., Goodale, P.: Selecting success criteria: experiences with an academic library catalogue. In: Forner et al. [138], pp. 59–70

    Google Scholar 

  83. Clough, P., Grubinger, M., Deselaers, T., Hanbury, A., Müller, H.: Overview of the ImageCLEF 2006 photographic retrieval and object annotation tasks. In: Peters et al. [357], pp. 223–256

    Google Scholar 

  84. Clough, P., et al.: The CLEF 2005 cross-language image retrieval track. In: Peters et al. [365], pp. 535–557

    Google Scholar 

  85. Clough, P., Müller, H., Sanderson, M.: The CLEF 2004 cross-language image retrieval track. In: Peters et al. [358], pp. 597–613

    Google Scholar 

  86. Clough, P., Sanderson, M.: The CLEF 2003 cross language image retrieval track. In: Peters et al. [356], pp. 581–593

    Google Scholar 

  87. Clough, P., Tsikrika, T.: Multi-lingual retrieval of pictures in ImageCLEF. In: Ferro and Peters [130]

    Google Scholar 

  88. Clough, P., Willett, P., Lim, J.: Unfair means: use cases beyond plagiarism. In: Mothe et al. [301], pp. 229–234

    Google Scholar 

  89. Coello-Guilarte, D.L., Ortega Mendoza, R.M., Villaseñor-Pineda, L., Montes-y Gómez, M.: Cross-lingual depression detection in Twitter using bilingual word-level alignment. In: Crestani et al. [94]

    Google Scholar 

  90. Collovini, S., Marcelo de Bairros Filho, P., Vieira, R.: Analysing the role of representation choices in portuguese relation extraction. In: Mothe et al. [301], pp. 105–116

    Google Scholar 

  91. Conlan, O., Fraser, K., Kelly, L., Yousuf, B.: A user modeling shared challenge proposal. In: Crestani et al. [94]

    Google Scholar 

  92. Corezola Pereira, R., Pereira Moreira, V., Galante, R.: A new approach for cross-language plagiarism analysis. In: Agosti et al. [9], pp. 15–26

    Chapter  Google Scholar 

  93. Cossu, J.V., Ferreira, E., Janod, K., Gaillard, J., El-Bèze, M.: NLP-based classifiers to generalize expert assessments in e-reputation. In: Mothe et al. [301], pp. 340–351

    Google Scholar 

  94. Crestani, F., et al. (eds.): CLEF 2019. LNCS, vol. 11696. Springer, Heidelberg (2019)

    Google Scholar 

  95. Custódio, J., Paraboni, I.: An ensemble approach to cross-domain authorship attribution. In: Crestani et al. [94]

    Google Scholar 

  96. Dadashkarimi, J., Esfahani, H.N., Faili, H., Shakery, A.: SS4MCT: a statistical stemmer for morphologically complex texts. In: Fuhr et al. [143], pp. 201–207

    Google Scholar 

  97. Daelemans, W., et al.: Overview of PAN 2019: bots and gender profiling, celebrity profiling, cross-domain authorship attribution and style change detection. In: Crestani et al. [94]

    Google Scholar 

  98. Dehghani, M., Azarbonyad, H., Kamps, J., Marx, M.: Two-way parsimonious classification models for evolving hierarchies. In: Fuhr et al. [143], pp. 69–82

    Google Scholar 

  99. Déjean, S., Mothe, J., Ullah, M.Z.: Studying the variability of system setting effectiveness by data analytics and visualization. In: Crestani et al. [94]

    Google Scholar 

  100. Deneu, B., Servajean, M., Botella, C., Joly, A.: Evaluation of deep species distribution models using environment and co-occurrences. In: Crestani et al. [94]

    Google Scholar 

  101. Deselaers, T., Deserno, T.M.: Medical image annotation in ImageCLEF 2008. In: Peters et al. [359], pp. 523–530

    Google Scholar 

  102. Deselaers, T., Hanbury, A.: The visual concept detection task in ImageCLEF 2008. In: Peters et al. [359], pp. 531–538

    Google Scholar 

  103. Deselaers, T., et al.: Overview of the ImageCLEF 2007 object retrieval task. In: Peters et al. [366], pp. 445–471

    Google Scholar 

  104. Devezas, J., Nunes, S.: Index-based semantic tagging for efficient query interpretation. In: Fuhr et al. [143], pp. 208–213

    Google Scholar 

  105. Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., Tino, G.: To Re-rank or to re-query: can visual analytics solve this dilemma? In: Forner et al. [136], pp. 119–130

    Google Scholar 

  106. Di Nunzio, G.M., Ferro, N., Jones, G.J.F., Peters, C.: CLEF 2005: ad hoc track overview. In: Peters et al. [365], pp. 11–36

    Google Scholar 

  107. Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2006: ad hoc track overview. In: Peters et al. [357], pp. 21–34

    Google Scholar 

  108. Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2007: ad hoc track overview. In: Peters et al. [366], pp. 13–32

    Google Scholar 

  109. Di Nunzio, G.M., Leveling, J., Mandl, T.: LogCLEF 2011 multilingual log file analysis: language identification, query classification, and success of a query. In: Petras et al. [373]

    Google Scholar 

  110. Di Nunzio, G.M., Vezzani, F.: Using R markdown for replicable experiments in evidence based medicine. In: Bellot et al. [42], pp. 28–39

    Google Scholar 

  111. Dicente Cid, Y., Batmanghelich, K., Müller, H.: Textured graph-based model of the lungs: application on tuberculosis type classification and multi-drug resistance detection. In: Bellot et al. [42], pp. 157–168

    Google Scholar 

  112. Dietz, F., Petras, V.: A component-level analysis of an academic search test collection. Part I: system and collection configurations. In: Jones et al. [209], pp. 16–28

    Google Scholar 

  113. Dietz, F., Petras, V.: A component-level analysis of an academic search test collection.- Part II: query analysis. In: Jones et al. [209], pp. 29–42

    Google Scholar 

  114. Domann, J., Lommatzsch, A.: A highly available real-time news recommender based on apache spark. In: Jones et al. [209], pp. 161–172

    Google Scholar 

  115. Elsayed, T., et al.: Overview of the CLEF-2019 CheckThat!: automatic identification and verification of claims. In: Crestani et al. [94]

    Google Scholar 

  116. Ermakova, K.: A method for short message contextualization: experiments at CLEF/INEX. In: Mothe et al. [301], pp. 352–363

    Google Scholar 

  117. Ermakova, L., Goeuriot, L., Mothe, J., Mulhem, P., Nie, J.Y., SanJuan, E.: CLEF 2017 microblog cultural contextualization lab overview. In: Jones et al. [209], pp. 304–314

    Google Scholar 

  118. Esuli, F., Sebastiani, F.: Evaluating information extraction. In: Agosti et al. [9], pp. 100–111

    Chapter  Google Scholar 

  119. Ezzeldin, A.M., Kholief, M.H., El-Sonbaty, Y.: ALQASIM: Arabic language question answer selection in machines. In: Forner et al. [138], pp. 100–103

    Google Scholar 

  120. Federico, M., Bertoldi, N., Levow, G.A., Jones, G.J.F.: CLEF 2004 cross-language spoken document retrieval track. In: Peters et al. [358], pp. 816–820

    Google Scholar 

  121. Federico, M., Jones, G.J.F.: The CLEF 2003 cross-language spoken document retrieval track. In: Peters et al. [356], p. 646

    Google Scholar 

  122. Ferrante, M., Ferro, N., Maistro, M.: Rethinking how to extend average precision to graded relevance. In: Kanoulas et al. [216], pp. 19–30

    Google Scholar 

  123. Ferro, N.: What happened in CLEF... for a while? In: Crestani et al. [94]

    Google Scholar 

  124. Ferro, N., et al.: Manifesto from Dagstuhl Perspectives Workshop 17442 - From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences, vol. 7, no. 1, pp. 96–139. Dagstuhl Manifestos, Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany (2018)

    Google Scholar 

  125. Ferro, N., Fuhr, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF 2019. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds.) ECIR 2019, Part II. LNCS, vol. 11438, pp. 283–290. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15719-7_38

    Chapter  Google Scholar 

  126. Ferro, N., Harman, D.: CLEF 2009: Grid@CLEF pilot track overview. In: Peters et al. [360], pp. 552–565

    Google Scholar 

  127. Ferro, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF2018: overview of the replicability task. In: Cappellato et al. [71]

    Google Scholar 

  128. Ferro, N., Peters, C.: CLEF 2009 ad hoc track overview: TEL & Persian tasks. In: Peters et al. [360], pp. 13–35

    Google Scholar 

  129. Ferro, N., Peters, C.: From multilingual to multimodal: the evolution of CLEF over two decades. In: Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF [130]

    Google Scholar 

  130. Ferro, N., Peters, C.: Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF. The Information Retrieval Series, vol. 41. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22948-1

    Book  Google Scholar 

  131. Ferro, N., Santucci, G.: Visual analytics and IR experimental evaluation. In: Ferro and Peters [130]

    Google Scholar 

  132. Ferro, N., Silvello, G.: CLEF 15th birthday: what can we learn from ad hoc retrieval? In: Kanoulas et al. [216], pp. 31–43

    Google Scholar 

  133. Ferro, N., Silvello, G.: The CLEF monolingual grid of points. In: Fuhr et al. [143], pp. 16–27

    Google Scholar 

  134. Ferro, N., Silvello, G.: 3.5K runs, 5K topics, 3M assessments and 70M measures: what trends in 10 years of Adhoc-ish CLEF? Inf. Process. Manag. 53(1), 175–202 (2017)

    Article  Google Scholar 

  135. Fontanella, S., Rodríguez-Sánchez, A.J., Piater, J., Szedmak, S.: Kronecker decomposition for image classification. In: Fuhr et al. [143], pp. 137–149

    Google Scholar 

  136. Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.): CLEF 2011. LNCS, vol. 6941. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23708-9

    Book  Google Scholar 

  137. Forner, P., Karlgren, J., Womser-Hacker, C., Ferro, N. (eds.): CLEF 2012 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2012). ISSN 1613-0073. http://ceur-ws.org/Vol-1178/

  138. Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.): CLEF 2013. LNCS, vol. 8138. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40802-1

    Book  Google Scholar 

  139. Forner, P., Navigli, R., Tufis, D., Ferro, N. (eds.): CLEF 2013 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2013). ISSN 1613-0073. http://ceur-ws.org/Vol-1179/

  140. Forner, P., et al.: Overview of the Clef 2008 Multilingual Question Answering Track. In: Peters et al. [359], pp. 262–295

    Google Scholar 

  141. Franco-Salvador, M., Rangel Pardo, F.M., Rosso, P., Taulé, M., Martí, M.A.: Language variety identification using distributed representations of words and documents. In: Mothe et al. [301], pp. 28–40

    Google Scholar 

  142. Fuhr, N.: Reproducibility and validity in CLEF. In: Ferro and Peters [130]

    Google Scholar 

  143. Fuhr, N., et al. (eds.): CLEF 2016. LNCS, vol. 9822. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44564-9

    Book  Google Scholar 

  144. Gäde, M., Ferro, N., Lestari Paramita, M.: CHiC 2011 - cultural heritage in CLEF: from use cases to evaluation in practice for multilingual information access to cultural heritage. In: Petras et al. [373]

    Google Scholar 

  145. Gäde, M., Stiller, J., Petras, V.: Which log for which information? Gathering multilingual data from different log file types. In: Agosti et al. [9], pp. 70–81

    Google Scholar 

  146. Galuscáková, P., Pecina, P., Hajic, J.: Penalty functions for evaluation measures of unsegmented speech retrieval. In: Catarci et al. [75], pp. 100–111

    Google Scholar 

  147. Ganguly, D., Jones, G.J.F.: A gamified approach to relevance judgement. In: Fuhr et al. [143], pp. 214–220

    Google Scholar 

  148. Ganguly, D., Leveling, J., Jones, G.J.F.: Simulation of within-session query variations using a text segmentation approach. In: Forner et al. [136], pp. 89–94

    Google Scholar 

  149. Ganguly, D., Leveling, J., Jones, G.J.F.: A case study in decompounding for Bengali information retrieval. In: Forner et al. [138], pp. 108–119

    Google Scholar 

  150. Gebremeskel, G.G., de Vries, A.P.: Random performance differences between online recommender system algorithms. In: Fuhr et al. [143], pp. 187–200

    Google Scholar 

  151. Gey, F., et al.: GeoCLEF 2006: the CLEF 2006 cross-language geographic information retrieval track overview. In: Peters et al. [357], pp. 852–876

    Google Scholar 

  152. Gey, F.C., Larson, R.R., Sanderson, M., Joho, H., Clough, P., Petras, V.: GeoCLEF: the CLEF 2005 cross-language geographic information retrieval track overview. In: Peters et al. [365], pp. 908–919

    Google Scholar 

  153. Ghosh, S., Singhania, P., Singh, S., Rudra, K., Ghosh, S.: Stance detection in web and social media: a comparative study. In: Crestani et al. [94]

    Google Scholar 

  154. Giampiccolo, D., et al.: Overview of the CLEF 2007 multilingual question answering track. In: Peters et al. [366], pp. 200–236

    Google Scholar 

  155. Gînsca, A.L., Popescu, A., Lupu, M., Iftene, A., Kanellos, I.: Evaluating user image tagging credibility. In: Mothe et al. [301], pp. 41–52

    Google Scholar 

  156. Glinos, D.G.: Discovering similar passages within large text documents. In: Kanoulas et al. [216], pp. 98–109

    Google Scholar 

  157. Gobeill, J., Gaudinat, A., Ruch, P.: Instance-based learning for tweet monitoring and categorization. In: Mothe et al. [301], pp. 235–240

    Google Scholar 

  158. Goëau, H., et al.: The CLEF 2011 plant images classification task. In: Petras et al. [373]

    Google Scholar 

  159. Goëau, H., et al.: The ImageCLEF 2012 plant identification task. In: Forner et al. [137]

    Google Scholar 

  160. Goeuriot, L., et al.: Overview of the CLEF eHealth evaluation lab 2015. In: Mothe et al. [301], pp. 429–443

    Google Scholar 

  161. Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones et al. [209], pp. 291–303

    Google Scholar 

  162. Goeuriot, L., Mothe, J., Mulhem, P., Murtagh, F., SanJuan, E.: Overview of the CLEF 2016 cultural micro-blog contextualization workshop. In: Fuhr et al. [143], pp. 371–378

    Google Scholar 

  163. Gollub, T., et al.: Recent trends in digital text forensics and its evaluation - plagiarism detection, author identification, and author profiling. In: Forner et al. [138], pp. 282–302

    Google Scholar 

  164. Gómez-Adorno, H., Martín-del Campo-Rodríguez, C., Sidorov, G., Alemán, Y., Vilariño, D., Pinto, D.: Hierarchical clustering analysis: the best-performing approach at PAN 2017 author clustering task. In: Bellot et al. [42], pp. 216–223

    Google Scholar 

  165. Gonzalo, J., Clough, P., Karlgren, J.: Overview of iCLEF 2008: search log analysis for multilingual image retrieval. In: Peters et al. [359], pp. 227–235

    Google Scholar 

  166. Gonzalo, J., Clough, P., Vallin, A.: Overview of the CLEF 2005 interactive track. In: Peters et al. [365], pp. 251–262

    Google Scholar 

  167. Gonzalo, J., Oard, D.W.: The CLEF 2002 interactive track. In: Peters et al. [355], pp. 372–382

    Chapter  Google Scholar 

  168. Gonzalo, J., Oard, D.W.: iCLEF 2004 track overview: pilot experiments in interactive cross-language question answering. In: Peters et al. [358], pp. 310–322

    Google Scholar 

  169. Gonzalo, J., Peinado, V., Clough, P., Karlgren, J.: Overview of iCLEF 2009: exploring search behaviour in a multilingual folksonomy environment. In: Peters et al. [368], pp. 13–20

    Google Scholar 

  170. Goodwin, T., Harabagiu, S.M.: The impact of belief values on the identification of patient cohorts. In: Forner et al. [138], pp. 155–166

    Google Scholar 

  171. Grotov, A., Chuklin, A., Markov, I., Stout, L., Xumara, F., de Rijke, M.: A comparative study of click models for web search. In: Mothe et al. [301], pp. 78–90

    Google Scholar 

  172. Grubinger, M., Clough, P., Hanbury, A., Müller, H.: Overview of the ImageCLEFphoto 2007 photographic retrieval task. In: Peters et al. [366], pp. 433–444

    Google Scholar 

  173. Gupta, P., Barrón-Cedeño, A., Rosso, P.: Cross-language high similarity search using a conceptual thesaurus. In: Catarci et al. [75], pp. 67–75

    Chapter  Google Scholar 

  174. Hagen, M., Glimm, C.: Supporting more-like-this information needs: finding similar web content in different scenarios. In: Kanoulas et al. [216], pp. 50–61

    Google Scholar 

  175. Hall, M., Toms, E.: Building a common framework for IIR evaluation. In: Forner et al. [138], pp. 17–28

    Google Scholar 

  176. Halvani, O., Graner, L.: Rethinking the evaluation methodology of authorship verification methods. In: Bellot et al. [42], pp. 40–51

    Google Scholar 

  177. Hammarström, H.: Automatic annotation of bibliographical references for descriptive language materials. In: Forner et al. [136], pp. 62–73

    Chapter  Google Scholar 

  178. Han Lee, S., Loong Chang, Y., Seng Chan, S., Alexis, J., Bonnet, P., Goëau, H.: Plant classification based on gated recurrent unit. In: Bellot et al. [42], pp. 169–180

    Google Scholar 

  179. Hanbury, A., Müller, H.: Automated component-level evaluation: present and future. In: Agosti et al. [9], pp. 124–135

    Google Scholar 

  180. Hanbury, A., Müller, H., Langs, G., Weber, M.A., Menze, B.H., Salas Fernandez, T.: Bringing the algorithms to the data: cloud-based benchmarking for medical image analysis. In: Catarci et al. [75], pp. 24–29

    Google Scholar 

  181. Harman, D.K.: Information Retrieval Evaluation. Morgan & Claypool Publishers, San Rafael (2011)

    Book  Google Scholar 

  182. Harman, D.K., Voorhees, E.M. (eds.): TREC. Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)

    Google Scholar 

  183. Harris, C.G., Xu, T.: The importance of visual context clues in multimedia translation. In: Forner et al. [136], pp. 107–118

    Google Scholar 

  184. Hasan, S.A., et al.: Attention-based medical caption generation with image modality classification and clinical concept mapping. In: Bellot et al. [42], pp. 224–230

    Google Scholar 

  185. Hiemstra, D., Hauff, C.: MapReduce for information retrieval evaluation: “let’s quickly test this on 12 TB of data”. In: Agosti et al. [9], pp. 64–69

    Google Scholar 

  186. Hoang, T.B.N., Mothe, J., Baillon, M.: TwitCID: a collection of data sets for studies on information diffusion on social networks. In: Crestani et al. [94]

    Google Scholar 

  187. Hopfgartner, F., Balog, K., Lommatzsch, A., Kelly, L., Kille, B., Schuth, A., Larson, M.: Continuous evaluation of large-scale information access systems: a case for living labs. In: Ferro and Peters [130]

    Google Scholar 

  188. Hopfgartner, F., Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T., Heintz, T.: Benchmarking news recommendations in a living lab. In: Kanoulas et al. [216], pp. 250–267

    Google Scholar 

  189. Hull, D.A., Oard, D.W.: Cross-Language Text and Speech Retrieval - Papers from the AAAI Spring Symposium. Association for the Advancement of Artificial Intelligence (AAAI), Technical report SS-97-05 (1997). http://www.aaai.org/Press/Reports/Symposia/Spring/ss-97-05.php

  190. Hürriyetoğlu, A., et al.: Overview of CLEF 2019 lab ProtestNews: extracting protests from news in a cross-context setting. In: Crestani et al. [94]

    Google Scholar 

  191. Huurnink, B., Hofmann, K., de Rijke, M., Bron, M.: Validating query simulators: an experiment using commercial searches and purchases. In: Agosti et al. [9], pp. 40–51

    Google Scholar 

  192. Imhof, M., Braschler, M.: Are test collections “real”? Mirroring real-world complexity in IR test collections. In: Mothe et al. [301], pp. 241–247

    Google Scholar 

  193. Inches, G., Crestani, F.: Overview of the international sexual predator identification competition at PAN-2012. In: Forner et al. [137]

    Google Scholar 

  194. Ionescu, B., et al.: ImageCLEF 2019: multimedia retrieval in medicine, lifelogging, security and nature. In: Crestani et al. [94]

    Google Scholar 

  195. Ionescu, B., et al.: Overview of ImageCLEF 2017: information extraction from images. In: Jones et al. [209], pp. 315–337

    Google Scholar 

  196. Ionescu, B., et al.: Overview of ImageCLEF 2018: challenges, datasets and evaluation. In: Bellot et al. [42], pp. 309–334

    Google Scholar 

  197. Jabeur, L.B., Soulier, L., Tamine, L., Mousset, P.: A product feature-based user-centric ranking model for e-commerce search. In: Fuhr et al. [143], pp. 174–186

    Google Scholar 

  198. Jijkoun, V., de Rijke, M.: Overview of WebCLEF 2007. In: Peters et al. [366], pp. 725–731

    Google Scholar 

  199. Jijkoun, V., de Rijke, M.: Overview of WebCLEF 2008. In: Peters et al. [359], pp. 787–793

    Google Scholar 

  200. Joly, A., et al.: Overview of LifeCLEF 2018: a large-scale evaluation of species identification and recommendation algorithms in the era of AI. In: Bellot et al. [42], pp. 247–266

    Google Scholar 

  201. Joly, A., et al.: Overview of LifeCLEF 2019: identification of Amazonian plants, south & north American birds, and niche prediction. In: Crestani et al. [94]

    Google Scholar 

  202. Joly, A., et al.: LifeCLEF 2017 lab overview: multimedia species identification challenges. In: Jones et al. [209], pp. 255–274

    Google Scholar 

  203. Joly, A., et al.: Biodiversity information retrieval through large scale content-based identification: a long-term evaluation. In: Ferro and Peters [130]

    Google Scholar 

  204. Joly, A., et al.: LifeCLEF 2014: multimedia life species identification challenges. In: Kanoulas et al. [216], pp. 229–249

    Google Scholar 

  205. Joly, A., et al.: LifeCLEF 2015: multimedia life species identification challenges. In: Mothe et al. [301], pp. 462–483

    Google Scholar 

  206. Joly, A., et al.: LifeCLEF 2016: multimedia life species identification challenges. In: Fuhr et al. [143], pp. 286–310

    Google Scholar 

  207. Jones, G.J.F.: ’Bout sound and vision: CLEF beyond text retrieval tasks. In: Ferro and Peters [130]

    Google Scholar 

  208. Jones, G.J.F., Federico, M.: CLEF 2002 cross-language spoken document retrieval pilot track report. In: Peters et al. [355], pp. 446–457

    Google Scholar 

  209. Jones, G.J.F., et al. (eds.): CLEF 2017. LNCS, vol. 10456. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1

    Book  Google Scholar 

  210. Juola, P.: An overview of the traditional authorship attribution subtask. In: Forner et al. [137]

    Google Scholar 

  211. Jürgens, J., Hansen, P., Womser-Hacker, C.: Going beyond CLEF-IP: the ‘Reality’ for patent searchers. In: Catarci et al. [75], pp. 30–35

    Google Scholar 

  212. Kalpathy-Cramer, J., Müller, H., Bedrick, S., Eggel, I., Garcia Seco de Herrera, A., Tsikrika, T.: Overview of the CLEF 2011 medical image classification and retrieval tasks. In: Petras et al. [373]

    Google Scholar 

  213. Kamps, J., Koolen, M., Geva, S., Schenkel, R., SanJuan, E., Bogers, T.: From XML retrieval to semantic search and beyond. In: Ferro and Peters [130]

    Google Scholar 

  214. Kanoulas, E., Azzopardi, L.: CLEF 2017 dynamic search evaluation lab overview. In: Jones et al. [209], pp. 361–366

    Google Scholar 

  215. Kanoulas, E., Azzopardi, L., Hui Yang, G.: Overview of the CLEF dynamic search evaluation lab 2018. In: Bellot et al. [42], pp. 362–371

    Google Scholar 

  216. Kanoulas, E., et al. (eds.): CLEF 2014. LNCS, vol. 8685. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11382-1

    Book  Google Scholar 

  217. Karadzhov, G., Mihaylova, T., Kiprov, Y., Georgiev, G., Koychev, Y., Nakov, P.: The case for being average: a mediocrity approach to style masking and author obfuscation. In: Jones et al. [209], pp. 173–185

    Google Scholar 

  218. Karan, M., Snajder, J.: Evaluation of manual query expansion rules on a domain specific FAQ collection. In: Mothe et al. [301], pp. 248–253

    Google Scholar 

  219. Karisani, P., Oroumchian, F., Rahgozar, M.: Tweet expansion method for filtering task in Twitter. In: Mothe et al. [301], pp. 55–64

    Google Scholar 

  220. Karlgren, J.: Adopting systematic evaluation benchmarks in operational settings. In: Ferro and Peters [130]

    Google Scholar 

  221. Karlgren, J.: How lexical gold standards have effects on the usefulness of text analysis tools for digital scholarship. In: Crestani et al. [94]

    Google Scholar 

  222. Karlgren, J., et al.: Evaluating learning language representations. In: Mothe et al. [301], pp. 254–260

    Google Scholar 

  223. Karlgren, J., Gonzalo, J., Clough, P.: iCLEF 2006 overview: searching the Flickr WWW photo-sharing repository. In: Peters et al. [357], pp. 186–194

    Google Scholar 

  224. Karlgren, J., Hedlund, T., Järvelin, K., Keskustalo, H., Kettunen, K.: The challenges of language variation in information access. In: Ferro and Peters [130]

    Google Scholar 

  225. Karlsson, V., Herman, P., Karlgren, J.: Evaluating categorisation in real life - an argument against simple but impractical metrics. In: Fuhr et al. [143], pp. 221–226

    Google Scholar 

  226. Kavallieratou, E., del Blanco, C.R., Cuevas, C., García, N.: Interactive learning-based retrieval technique for visual lifelogging. In: Crestani et al. [94]

    Google Scholar 

  227. Kelly, L., Goeuriot, L., Suominen, H., Névéol, A., Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2016. In: Fuhr et al. [143], pp. 255–266

    Google Scholar 

  228. Kelly, L., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Kanoulas et al. [216], pp. 172–191

    Google Scholar 

  229. Keszler, A., Kovács, L., Szirányi, T.: The appearance of the giant component in descriptor graphs and its application for descriptor selection. In: Catarci et al. [75], pp. 76–81

    Google Scholar 

  230. Kharazmi, S., Scholer, F., Vallet, D., Sanderson, M.: Examining additivity and weak baselines. ACM Trans. Inf. Syst. (TOIS) 34(4), 23:1–23:18 (2016)

    Article  Google Scholar 

  231. Khwileh, A., Ganguly, D., Jones, G.J.F.: An investigation of cross-language information retrieval for user-generated internet video. In: Mothe et al. [301], pp. 117–129

    Google Scholar 

  232. Khwileh, A., Way, A., Jones, G.J.F.: Improving the reliability of query expansion for user-generated speech retrieval using query performance prediction. In: Jones et al. [209], pp. 43–56

    Google Scholar 

  233. Kille, B., et al.: Overview of NewsREEL’16: multi-dimensional evaluation of real-time stream-recommendation algorithms. In: Fuhr et al. [143], pp. 311–331

    Google Scholar 

  234. Kille, B., et al.: Stream-based recommendations: online and offline evaluation as a service. In: Mothe et al. [301], pp. 497–517

    Google Scholar 

  235. Kim, S.J., Lee, J.H.: Subtopic mining based on head-modifier relation and co-occurrence of intents using web documents. In: Forner et al. [138], pp. 179–191

    Google Scholar 

  236. Kliegr, T., Kuchar, J.: Benchmark of rule-based classifiers in the news recommendation task. In: Mothe et al. [301], pp. 130–141

    Google Scholar 

  237. Kluck, M.: The domain-specific track in CLEF 2004: overview of the results and remarks on the assessment process. In: Peters et al. [358], pp. 260–270

    Google Scholar 

  238. Kluck, M., Gey, F.C.: The domain-specific task of CLEF - specific evaluation strategies in cross-language information retrieval. In: Peters [345], pp. 48–56

    Chapter  Google Scholar 

  239. Kluck, M., Stempfhuber, M.: Domain-specific track CLEF 2005: overview of results and approaches, remarks on the assessment analysis. In: Peters et al. [365], pp. 212–221

    Google Scholar 

  240. Kocher, M., Savoy, J.: Author clustering with an adaptive threshold. In: Jones et al. [209], pp. 186–198

    Google Scholar 

  241. Koitka, S., Friedrich, C.M.: Optimized convolutional neural network ensembles for medical subfigure classification. In: Jones et al. [209], pp. 57–68

    Google Scholar 

  242. Koolen, M., et al.: Overview of the CLEF 2016 social book search lab. In: Fuhr et al. [143], pp. 351–370

    Google Scholar 

  243. Koolen, M., et al.: Overview of the CLEF 2015 social book search lab. In: Mothe et al. [301], pp. 545–564

    Google Scholar 

  244. Koolen, M., Kazai, G., Kamps, J., Preminger, M., Doucet, A., Landoni, M.: Overview of the INEX 2012 social book search track. In: Forner et al. [137]

    Google Scholar 

  245. Koops, H.V., Van Balen, J., Wiering, F.: Automatic segmentation and deep learning of bird sounds. In: Mothe et al. [301], pp. 261–267

    Google Scholar 

  246. Kordjamshidi, P., Rahgooy, T., Moens, M.F., Pustejovsky, J., Manzoor, U., Roberts, K.: CLEF 2017: multimodal spatial role labeling (mSpRL) task overview. In: Jones et al. [209], pp. 367–376

    Google Scholar 

  247. Kosmopoulos, A., Paliouras, G., Androutsopoulos, I.: The effect of dimensionality reduction on large scale hierarchical classification. In: Kanoulas et al. [216], pp. 160–171

    Google Scholar 

  248. Kumar, N.K., Santosh, G.S.K., Varma, V.: A language-independent approach to identify the named entities in under-resourced languages and clustering multilingual documents. In: Forner et al. [136], pp. 74–82

    Google Scholar 

  249. Kurimo, M., Creutz, M., Varjokallio, M.: Morpho challenge evaluation using a linguistic gold standard. In: Peters et al. [366], pp. 864–872

    Google Scholar 

  250. Kurimo, M., Turunen, V.T., Varjokallio, M.: Overview of morpho challenge 2008. In: Peters et al. [359], pp. 951–966

    Google Scholar 

  251. Kurimo, M., Virpioja, S., Turunen, V.T., Blackwood, G.W., Byrne, W.: Overview and results of morpho challenge 2009. In: Peters et al. [360], pp. 587–597

    Google Scholar 

  252. Kürsten, J., Eibl, M.: Comparing IR system components using beanplots. In: Catarci et al. [75], pp. 136–137

    Chapter  Google Scholar 

  253. Kvist, M., Velupillai, S.: SCAN: a swedish clinical abbreviation normalizer - further development and adaptation to radiology. In: Kanoulas et al. [216], pp. 62–73

    Google Scholar 

  254. de Pertile, S.L., Pereira Moreira, V.: A test collection to evaluate plagiarism by missing or incorrect references. In: Catarci et al. [75], pp. 141–143

    Google Scholar 

  255. de Pertile, S.L., Rosso, P., Pereira Moreira, V.: Counting co-occurrences in citations to identify plagiarised text fragments. In: Forner et al. [138], pp. 150–154

    Google Scholar 

  256. Lagopoulos, A., Anagnostou, A., Minas, A., Tsoumakas, G.: Learning-to-rank and relevance feedback for literature appraisal in empirical medicine. In: Bellot et al. [42], pp. 52–63

    Google Scholar 

  257. Lai, M., Tambuscio, M., Patti, V., Ruffo, G., Rosso, P.: Extracting graph topological information and users’ opinion. In: Jones et al. [209], pp. 112–118

    Google Scholar 

  258. Landoni, M., Matteri, D., Murgia, E., Huibers, T., Soledad Pera, M.: Sonny, Cerca! evaluating the impact of using a vocal assistant to search at school. In: Crestani et al. [94]

    Google Scholar 

  259. Larsen, B.: The scholarly impact of CLEF 2010–2017. In: Ferro and Peters [130]

    Google Scholar 

  260. Larson, M., Newman, E., Jones, G.J.F.: Overview of VideoCLEF 2008: automatic generation of topic-based feeds for dual language audio-visual content. In: Peters et al. [359], pp. 906–917

    Google Scholar 

  261. Larson, M., Newman, E., Jones, G.J.F.: Overview of VideoCLEF 2009: new perspectives on speech-based multimedia content enrichment. In: Peters et al. [368], pp. 354–368

    Google Scholar 

  262. Lasseck, M.: Towards automatic large-scale identification of birds in audio recordings. In: Mothe et al. [301], pp. 364–375

    Google Scholar 

  263. Leiva, L.A., Villegas, M., Paredes, R.: Relevant clouds: leveraging relevance feedback to build tag clouds for image search. In: Forner et al. [138], pp. 143–149

    Google Scholar 

  264. Leong, C.W., Hassan, S., Ruiz, M.E., Rada, M.: Improving query expansion for image retrieval via saliency and picturability. In: Forner et al. [136], pp. 137–142

    Google Scholar 

  265. Lestari Paramita, M., Sanderson, M., Clough, P.: Diversity in photo retrieval: overview of the ImageCLEFPhoto task 2009. In: Peters et al. [368], pp. 45–59

    Google Scholar 

  266. Li, P., Jiang, X., Kambhamettu, C., Shatkay, H.: Segmenting compound biomedical figures into their constituent panels. In: Jones et al. [209], pp. 199–210

    Google Scholar 

  267. Li, W., Jones, G.J.F.: Enhancing medical information retrieval by exploiting a content-based recommender method. In: Mothe et al. [301], pp. 142–153

    Google Scholar 

  268. Linhares Pontes, E., Huet, S., Torres-Moreno, J.M.: Microblog contextualization: advantages and limitations of a multi-sentence compression approach. In: Bellot et al. [42], pp. 181–190

    Google Scholar 

  269. Lipani, A., Piroi, F., Andersson, L., Hanbury, A.: An information retrieval ontology for information retrieval nanopublications. In: Kanoulas et al. [216], pp. 44–49

    Google Scholar 

  270. Litvinova, T., Seredin, P., Litvinova, O., Ryzhkova, E.: Estimating the similarities between texts of right-handed and left-handed males and females. In: Jones et al. [209], pp. 119–124

    Google Scholar 

  271. Liu, F., Peng, Y., Rosen, M.P.: An effective deep transfer learning and information fusion framework for medical visual question answering. In: Crestani et al. [94]

    Google Scholar 

  272. Lommatzsch, A., et al.: CLEF 2017 NewsREEL overview: a stream-based recommender task for evaluation and education. In: Jones et al. [209], pp. 239–254

    Google Scholar 

  273. Lommatzsch, A., Werner, S.: Optimizing and evaluating stream-based news recommendation algorithms. In: Mothe et al. [301], pp. 376–388

    Google Scholar 

  274. Loponen, A., Järvelin, K.: A dictionary- and corpus-independent statistical lemmatizer for information retrieval in low resource languages. In: Agosti et al. [9], pp. 3–14

    Google Scholar 

  275. Losada, D.E., Crestani, F.: A test collection for research on depression and language use. In: Fuhr et al. [143], pp. 28–39

    Google Scholar 

  276. Losada, D.E., Crestani, F., Parapar, J.: eRISK 2017: CLEF lab on early risk prediction on the internet: experimental foundations. In: Jones et al. [209], pp. 346–360

    Google Scholar 

  277. Losada, D.E., Crestani, F., Parapar, J.: Overview of eRisk: early risk prediction on the internet. In: Bellot et al. [42], pp. 343–361

    Google Scholar 

  278. Losada, D.E., Crestani, F., Parapar, J.: Overview of eRisk 2019: early risk prediction on the internet. In: Crestani et al. [94]

    Google Scholar 

  279. Mackie, S., McCreadie, R., Macdonald, C., Ounis, I.: Comparing algorithms for microblog summarisation. In: Kanoulas et al. [216], pp. 153–159

    Google Scholar 

  280. Magdy, W., Jones, G.J.F.: Examining the robustness of evaluation metrics for patent retrieval with incomplete relevance judgements. In: Agosti et al. [9], pp. 82–93

    Google Scholar 

  281. Magnini, B., et al.: Overview of the CLEF 2006 multilingual question answering track. In: Peters et al. [357], pp. 223–256

    Google Scholar 

  282. Magnini, B., et al.: The multiple language question answering track at CLEF 2003. In: Peters et al. [356], pp. 471–486

    Google Scholar 

  283. Magnini, B., et al.: Overview of the CLEF 2004 multilingual question answering track. In: Peters et al. [358], pp. 371–391

    Google Scholar 

  284. Mandl, T., et al.: LogCLEF 2009: the CLEF 2009 multilingual logfile analysis track overview. In: Peters et al. [360], pp. 508–517

    Google Scholar 

  285. Mandl, T., et al.: GeoCLEF 2008: the CLEF 2008 cross-language geographic information retrieval track overview. In: Peters et al. [359], pp. 808–821

    Google Scholar 

  286. Mandl, T., Di Nunzio, G.M., Schulz, J.M.: LogCLEF 2010: the CLEF 2010 multilingual logfile analysis track overview. In: Braschler et al. [61]

    Google Scholar 

  287. Mandl, T., et al.: GeoCLEF 2007: the CLEF 2007 cross-language geographic information retrieval track overview. In: Peters et al. [366], pp. 745–772

    Google Scholar 

  288. Manotumruksa, J., Macdonald, C., Ounis, I.: Predicting contextually appropriate venues in location-based social networks. In: Fuhr et al. [143], pp. 96–109

    Google Scholar 

  289. Martínez-Gómez, J., García-Varea, I., Caputo, B.: Overview of the ImageCLEF 2012 robot vision task. In: Forner et al. [137]

    Google Scholar 

  290. Mayfield, J., Lawrie, D., McNamee, P., Oard, D.W.: Building a cross-language entity linking collection in twenty-one languages. In: Forner et al. [136], pp. 3–13

    Google Scholar 

  291. McCreadie, R., Macdonald, C., Ounis, I., Brassey, J.: A study of personalised medical literature search. In: Kanoulas et al. [216], pp. 74–85

    Google Scholar 

  292. McMinn, A.J., Jose, J.M.: Real-time entity-based event detection for Twitter. In: Mothe et al. [301], pp. 65–77

    Google Scholar 

  293. Medvedeva, M., Haagsma, H., Nissim, M.: An analysis of cross-genre and in-genre performance for author profiling in social media. In: Jones et al. [209], pp. 211–223

    Google Scholar 

  294. Miftahutdinov, Z., Tutubalina, E.: Deep learning for ICD coding: looking for medical concepts in clinical documents in English and in French. In: Bellot et al. [42], pp. 203–215

    Google Scholar 

  295. Mirsarraf, M.R., Dehghani, N.: A dependency-inspired semantic evaluation of machine translation systems. In: Forner et al. [138], pp. 71–74

    Google Scholar 

  296. Mitrovic, S., Müller, H.: Summarizing citation contexts of scientific publications. In: Mothe et al. [301], pp. 154–165

    Google Scholar 

  297. Molina, A., SanJuan, E., Torres-Moreno, J.M.: A Turing test to evaluate a complex summarization task. In: Forner et al. [138], pp. 75–80

    Google Scholar 

  298. Molina, S., Mothe, J., Roques, D., Tanguy, L., Ullah, M.Z.: IRIT-QFR: IRIT query feature resource. In: Jones et al. [209], pp. 69–81

    Google Scholar 

  299. Morante, R., Daelemans, W.: Overview of the QA4MRE pilot task: annotating modality and negation for a machine reading evaluation. In: Petras et al. [373]

    Google Scholar 

  300. Moreno, R., Huáng, W., Younus, A., O’Mahony, M.P., Hurley, N.J.: Evaluation of hierarchical clustering via markov decision processes for efficient navigation and search. In: Jones et al. [209], pp. 125–131

    Google Scholar 

  301. Mothe, J., et al. (eds.): CLEF 2015. LNCS, vol. 9283. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24027-5

    Book  Google Scholar 

  302. Mulhem, P., Goeuriot, L., Dogra, N., Amer, N.O.: TimeLine illustration based on microblogs: when diversification meets metadata re-ranking. In: Jones et al. [209], pp. 224–235

    Google Scholar 

  303. Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF - Experimental Evaluation in Visual Information Retrieval. Springer-Verlag, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15181-1

    Book  MATH  Google Scholar 

  304. Müller, H., Deselaers, T., Deserno, T.M., Clough, P., Kim, E., Hersh, W.R.: Overview of the ImageCLEFmed 2006 medical retrieval and medical annotation tasks. In: Peters et al. [357], pp. 595–608

    Google Scholar 

  305. Müller, H., Deselaers, T., Deserno, T.M., Kalpathy-Cramer, J., Kim, E., Hersh, W.R.: Overview of the ImageCLEFmed 2007 medical retrieval and medical annotation tasks. In: Peters et al. [366], pp. 472–491

    Google Scholar 

  306. Müller, H., Hanbury, A.: EaaS: evaluation-as-a-service and experiences from the VISCERAL project. In: Ferro and Peters [130]

    Google Scholar 

  307. Müller, H., Garcia Seco de Herrera, A., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S., Eggel, I.: Overview of the ImageCLEF 2012 medical image retrieval and classification tasks. In: Forner et al. [137]

    Google Scholar 

  308. Müller, H., et al.: Overview of the CLEF 2009 medical image retrieval track. In: Peters et al. [368], pp. 72–84

    Google Scholar 

  309. Müller, H., et al.: Overview of the CLEF 2010 medical image retrieval track. In: Braschler et al. [61]

    Google Scholar 

  310. Müller, H., Kalpathy-Cramer, J., Garcia Seco de Herrera, A.: Experiences from the ImageCLEF medical retrieval and annotation tasks. In: Ferro and Peters [130]

    Google Scholar 

  311. Müller, H., Kalpathy-Cramer, J., Kahn, C.E., Hatt, W., Bedrick, S., Hersh, W.: Overview of the ImageCLEFmed 2008 medical image retrieval task. In: Peters et al. [359], pp. 512–522

    Google Scholar 

  312. Murauer, B., Specht, G.: Generating cross-domain text corpora from social media comments. In: Crestani et al. [94]

    Google Scholar 

  313. Nakov, P., et al.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. In: Bellot et al. [42], pp. 372–387

    Google Scholar 

  314. Nardi, A., Peters, C., Ferro, N. (eds.): CLEF 2007 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2007). ISSN 1613-0073. http://ceur-ws.org/Vol-1173/

  315. Nardi, A., Peters, C., Vicedo, J.L., Ferro, N. (eds.): CLEF 2006 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2006). ISSN 1613-0073. http://ceur-ws.org/Vol-1172/

  316. Nordlie, R., Pharo, N.: Seven years of INEX interactive retrieval experiments - lessons and challenges. In: Catarci et al. [75], pp. 13–23

    Google Scholar 

  317. Nowak, S., Dunker, P.: Overview of the CLEF 2009 large-scale visual concept detection and annotation task. In: Peters et al. [368], pp. 94–109

    Google Scholar 

  318. Nowak, S., Huiskes, M.J.: New strategies for image annotation: overview of the photo annotation task at ImageCLEF 2010. In: Braschler et al. [61]

    Google Scholar 

  319. Nowak, S., Nagel, K., Liebetrau, J.: The CLEF 2011 photo annotation and concept-based retrieval tasks. In: Petras et al. [373]

    Google Scholar 

  320. Oard, D.W., Gonzalo, J.: The CLEF 2001 interactive track. In: Peters et al. [354], pp. 308–319

    Google Scholar 

  321. Oard, D.W., Gonzalo, J.: The CLEF 2003 interactive track. In: Peters et al. [356], pp. 425–434

    Google Scholar 

  322. Oard, D.W., et al.: Overview of the CLEF-2006 cross-language speech retrieval track. In: Peters et al. [357], pp. 744–758

    Google Scholar 

  323. Oh, H.S., Jung, Y., Kim, K.Y.: A multiple-stage approach to re-ranking medical documents. In: Mothe et al. [301], pp. 166–177

    Google Scholar 

  324. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Multilingual question-answering system in biomedical domain on the web: an evaluation. In: Forner et al. [136], pp. 83–88

    Google Scholar 

  325. Orio, N., Liem, C.C.S., Peeters, G., Schedl, M.: MusiClef: multimodal music tagging task. In: Catarci et al. [75], pp. 36–41

    Google Scholar 

  326. Orio, N., Rizo, D.: Overview of MusiCLEF 2011. In: Petras et al. [373]

    Google Scholar 

  327. Ortega-Mendoza, M., Franco-Arcega, A., López-Monroy, A.P., Montes-y Gómez, M.: I, me, mine: the role of personal phrases in author profiling. In: Fuhr et al. [143], pp. 110–122

    Google Scholar 

  328. Otterbacher, J.: Addressing social bias in information retrieval. In: Bellot et al. [42], pp. 121–127

    Google Scholar 

  329. Pääkkönen, T., et al.: Exploring behavioral dimensions in session effectiveness. In: Mothe et al. [301], pp. 178–189

    Google Scholar 

  330. Palotti, J., Zuccon, G., Bernhardt, J., Hanbury, A., Goeuriot, L.: Assessors agreement: a case study across assessor type, payment levels, query variations and relevance dimensions. In: Fuhr et al. [143], pp. 40–53

    Google Scholar 

  331. Parks, M., Karlgren, J., Stymne, S.: Plausibility testing for lexical resources. In: Jones et al. [209], pp. 132–137

    Google Scholar 

  332. Pasi, G., et al.: Evaluation of personalised information retrieval at CLEF 2018 (PIR-CLEF). In: Bellot et al. [42], pp. 335–342

    Google Scholar 

  333. Pasi, G., Jones, G.J.F., Goeuriot, L., Kelly, L., Marrara, S., Sanvitto, C.: Overview of the CLEF 2019 personalised information retrieval lab (PIR-CLEF 2019). In: Crestani et al. [94]

    Google Scholar 

  334. Pasi, G., Jones, G.J.F., Marrara, S., Sanvitto, C., Ganguly, D., Sen, P.: Overview of the CLEF 2017 personalised information retrieval pilot lab (PIR-CLEF 2017). In: Jones et al. [209], pp. 338–345

    Google Scholar 

  335. Pecina, P., Hoffmannová, P., Jones, G.J.F., Zhang, Y., Oard, D.W.: Overview of the CLEF-2007 cross-language speech retrieval track. In: Peters et al. [366], pp. 674–686

    Google Scholar 

  336. Pellegrin, L., et al.: A two-step retrieval method for image captioning. In: Fuhr et al. [143], pp. 150–161

    Google Scholar 

  337. Peñas, A., Forner, P., Rodrigo, A., Sutcliffe, R.F.E., Forascu, C., Mota, C.: Overview of ResPubliQA 2010: question answering evaluation over European legislation. In: Braschler et al. [61]

    Google Scholar 

  338. Peñas, A., et al.: Overview of ResPubliQA 2009: question answering evaluation over european legislation. In: Peters et al. [360], pp. 174–196

    Google Scholar 

  339. Peñas, A., et al.: Overview of QA4MRE at CLEF 2011: question answering for machine reading evaluation. In: Petras et al. [373]

    Google Scholar 

  340. Peñas, A., Hovy, E.H., Forner, P., Rodrigo, A., Sutcliffe, R.F.E., Morante, R.: QA4MRE 2011–2013: overview of question answering for machine reading evaluation. In: Forner et al. [138], pp. 303–320

    Google Scholar 

  341. Peñas, A., et al.: Overview of QA4MRE at CLEF 2012: question answering for machine reading evaluation. In: Forner et al. [137]

    Google Scholar 

  342. Peñas, A., et al.: Results and lessons of the question answering track at CLEF. In: Ferro and Peters [130]

    Google Scholar 

  343. Peñas, A., Unger, C., Ngonga Ngomo, A.C.a.: Overview of CLEF question answering track 2014. In: Kanoulas et al. [216], pp. 300–306

    Google Scholar 

  344. Peñas, A., Unger, C., Paliouras, P., Kakadiaris, I.A.: Overview of the CLEF question answering track 2015. In: Mothe et al. [301], pp. 539–544

    Google Scholar 

  345. Peters, C. (ed.): CLEF 2000. LNCS, vol. 2069. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44645-1

    Book  MATH  Google Scholar 

  346. Peters, C.: Introduction. In: Cross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000) [345], pp. 1–6

    Google Scholar 

  347. Peters, C.: What happened in CLEF 2004? In: Peters et al. [358], pp. 1–9

    Google Scholar 

  348. Peters, C.: What happened in CLEF 2005? In: Peters et al. [365], pp. 1–10

    Google Scholar 

  349. Peters, C.: What happened in CLEF 2006? In: Peters et al. [357], pp. 1–10

    Chapter  Google Scholar 

  350. Peters, C.: What happened in CLEF 2007? In: Peters et al. [366], pp. 1–12

    Google Scholar 

  351. Peters, C.: What happened in CLEF 2008? In: Peters et al. [359], pp. 1–14

    Google Scholar 

  352. Peters, C.: What happened in CLEF 2009? In: Peters et al. [360], pp. 1–12

    Google Scholar 

  353. Peters, C., Braschler, M., Clough, P.: Multilingual Information Retrieval. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23008-0

    Book  Google Scholar 

  354. Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.): CLEF 2001. LNCS, vol. 2406. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45691-0

    Book  Google Scholar 

  355. Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.): CLEF 2002. LNCS, vol. 2785. Springer, Heidelberg (2003). https://doi.org/10.1007/b12018

    Book  Google Scholar 

  356. Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.): CLEF 2003. LNCS, vol. 3237. Springer, Heidelberg (2004). https://doi.org/10.1007/b102261

    Book  Google Scholar 

  357. Peters, C., et al. (eds.): CLEF 2006. LNCS, vol. 4730. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74999-8

    Book  Google Scholar 

  358. Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.): CLEF 2004. LNCS, vol. 3491. Springer, Heidelberg (2005). https://doi.org/10.1007/b138934

    Book  Google Scholar 

  359. Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, T., Peñas, A., Petras, V. (eds.): CLEF 2008. LNCS, vol. 5706. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04447-2

    Book  Google Scholar 

  360. Peters, C., Di Nunzio, G.M., Kurimo, M., Mandl, T., Mostefa, D., Peñas, A., Roda, G. (eds.): CLEF 2009. LNCS, vol. 6241. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15754-7

    Book  Google Scholar 

  361. Peters, C., Ferro, N. (eds.): CLEF 2000 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2000). ISSN 1613-0073. http://ceur-ws.org/Vol-1166/

  362. Peters, C., Ferro, N. (eds.): CLEF 2001 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2001). ISSN 1613-0073. http://ceur-ws.org/Vol-1167/

  363. Peters, C., Ferro, N. (eds.): CLEF 2002 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2002). ISSN 1613-0073. http://ceur-ws.org/Vol-1168/

  364. Peters, C., Ferro, N. (eds.): CLEF 2003 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2003). ISSN 1613-0073. http://ceur-ws.org/Vol-1169/

  365. Peters, C., et al. (eds.): CLEF 2005. LNCS, vol. 4022. Springer, Heidelberg (2006). https://doi.org/10.1007/11878773

    Book  Google Scholar 

  366. Peters, C., et al. (eds.): CLEF 2007. LNCS, vol. 5152. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85760-0

    Book  Google Scholar 

  367. Peters, C., Quochi, V., Ferro, N. (eds.): CLEF 2005 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2005). ISSN 1613-0073. http://ceur-ws.org/Vol-1171/

  368. Peters, C., et al. (eds.): CLEF 2009. LNCS, vol. 6242. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15751-6

    Book  Google Scholar 

  369. Petras, V., Baerisch, S.: The domain-specific track at CLEF 2008. In: Peters et al. [359], pp. 186–198

    Google Scholar 

  370. Petras, V., Baerisch, S., Stempfhuber, M.: The domain-specific track at CLEF 2007. In: Peters et al. [366], pp. 160–173

    Google Scholar 

  371. Petras, V., et al.: Cultural heritage in CLEF (CHiC) 2013. In: Forner et al. [138], pp. 192–211

    Google Scholar 

  372. Petras, V., et al.: Cultural heritage in CLEF (CHiC) overview 2012. In: Forner et al. [137]

    Google Scholar 

  373. Petras, V., Forner, P., Clough, P., Ferro, N. (eds.): CLEF 2011 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2011). ISSN 1613-0073. http://ceur-ws.org/Vol-1177/

  374. Piras, L., Caputo, B., Dang-Nguyen, D.T., Riegler, M., Halvorsen, P.: Image retrieval evaluation in specific domains. In: Ferro and Peters [130]

    Google Scholar 

  375. Piroi, F.: CLEF-IP 2010: retrieval experiments in the intellectual property domain. In: Braschler et al. [61]

    Google Scholar 

  376. Piroi, F., Hanbury, A.: Multilingual patent text retrieval evaluation: CLEF-IP. In: Ferro and Peters [130]

    Google Scholar 

  377. Piroi, F., Lupu, M., Hanbury, A.: Effects of language and topic size in patent IR: an empirical study. In: Catarci et al. [75], pp. 54–66

    Google Scholar 

  378. Piroi, F., Lupu, M., Hanbury, A.: Overview of CLEF-IP 2013 lab - information retrieval in the patent domain. In: Forner et al. [138], pp. 232–249

    Google Scholar 

  379. Piroi, F., Lupu, M., Hanbury, A., Sexton, A.P., Magdy, W., Filippov, I.V.: CLEF-IP 2012: retrieval experiments in the intellectual property domain. In: Forner et al. [137]

    Google Scholar 

  380. Piroi, F., Lupu, M., Hanbury, A., Zenz, V.: CLEF-IP 2011: retrieval in the intellectual property domain. In: Petras et al. [373]

    Google Scholar 

  381. Popescu, A., Tsikrika, T., Kludas, J.: Overview of the Wikipedia retrieval task at ImageCLEF 2010. In: Braschler et al. [61]

    Google Scholar 

  382. Potha, N., Stamatatos, E.: An improved impostors method for authorship verification. In: Jones et al. [209], pp. 138–144

    Google Scholar 

  383. Potthast, M., Barrón-Cedeño, A., Eiselt, A., Stein, B., Rosso, P.: Overview of the 2nd international competition on plagiarism detection. In: Braschler et al. [61]

    Google Scholar 

  384. Potthast, M., Eiselt, A., Barrón-Cedeño, A., Stein, B., Rosso, P.: Overview of the 3rd international competition on plagiarism detection. In: Petras et al. [373]

    Google Scholar 

  385. Potthast, M., et al.: Overview of the 4th international competition on plagiarism detection. In: Forner et al. [137]

    Google Scholar 

  386. Potthast, M., Gollub, T., Rangel Pardo, F., Rosso, P., Stamatatos, E., Stein, B.: Improving the reproducibility of PAN’s shared tasks: plagiarism detection, author identification, and author profiling. In: Kanoulas et al. [216], pp. 268–299

    Google Scholar 

  387. Potthast, M., Gollub, T., Wiegmann, M., Stein, b.: TIRA integrated research architecture. In: Ferro and Peters [130]

    Google Scholar 

  388. Potthast, M., Holfeld, T.: Overview of the 2nd international competition on Wikipedia vandalism detection. In: Petras et al. [373]

    Google Scholar 

  389. Potthast, M., Rangel Pardo, F.M., Tschuggnall, M., Stamatatos, E., Rosso, P., Stein, B.: Overview of PAN’17 - author identification, author profiling, and author obfuscation. In: Jones et al. [209], pp. 275–290

    Google Scholar 

  390. Potthast, M., Stein, B., Holfeld, T.: Overview of the 1st international competition on Wikipedia vandalism detection. In: Braschler et al. [61]

    Google Scholar 

  391. Pritsos, D.A., Stamatatos, E.: The impact of noise in web genre identification. In: Mothe et al. [301], pp. 268–273

    Google Scholar 

  392. Pronobis, A., Fornoni, M., Christensen, H.I., Caputo, B.: The Robot Vision Track at ImageCLEF 2010. In: Braschler et al. [61]

    Google Scholar 

  393. Pronobis, A., Xing, L., Caputo, B.: Overview of the CLEF 2009 robot vision track. In: Peters et al. [368], pp. 110–119

    Google Scholar 

  394. Raghavi, K.C., Chinnakotla, M.K., Black, A.W., Shrivastava, M.: WebShodh: a code mixed factoid question answering system for web. In: Jones et al. [209], pp. 104–111

    Google Scholar 

  395. Ragheb, W., Azé, J., Bringay, S., Servajean, M.: Language modeling in temporal mood variation models for early risk detection on the internet. In: Crestani et al. [94]

    Google Scholar 

  396. Rangel Pardo, F., Rosso, P.: On the multilingual and genre robustness of EmoGraphs for author profiling in social media. In: Mothe et al. [301], pp. 274–280

    Google Scholar 

  397. Rebholz-Schuhmann, D., et al.: Entity recognition in parallel multi-lingual biomedical corpora: the CLEF-ER laboratory overview. In: Forner et al. [138], pp. 353–367

    Google Scholar 

  398. Rekabsaz, N., Lupu, M.: A real-world framework for translator as expert retrieval. In: Kanoulas et al. [216], pp. 141–152

    Google Scholar 

  399. de Rijke, M., Balog, K., Bogers, T., van den Bosch, A.: On the evaluation of entity profiles. In: Agosti et al. [9], pp. 94–99

    Google Scholar 

  400. Roda, G., Tait, J., Piroi, F., Zenz, V.: CLEF-IP 2009: retrieval experiments in the intellectual property domain. In: Peters et al. [360], pp. 385–409

    Google Scholar 

  401. Rodrigo, A., Peñas, A., Verdejo, M.F.: Overview of the answer validation exercise 2008. In: Peters et al. [359], pp. 296–313

    Google Scholar 

  402. Roller, R., Stevenson, M.: Self-supervised relation extraction using UMLS. In: Kanoulas et al. [216], pp. 116–127

    Google Scholar 

  403. Rosso, P., et al.: Evolution of the PAN lab on digital text forensics. In: Ferro and Peters [130]

    Google Scholar 

  404. Rosso, P., Rangel, F., Potthast, M., Stamatatos, E., Tschuggnall, M., Stein, B.: Overview of PAN’16. In: Fuhr et al. [143], pp. 332–350

    Google Scholar 

  405. Rowe, B.R., Wood, D.W., Link, A.L., Simoni, D.A.: Economic Impact Assessment of NIST’s Text REtrieval Conference (TREC) Program. RTI Project Number 0211875, RTI International, USA, July 2010. http://trec.nist.gov/pubs/2010.economic.impact.pdf

  406. Sabetghadam, S., Bierig, R., Rauber, A.: A hybrid approach for multi-faceted IR in multimodal domain. In: Kanoulas et al. [216], pp. 86–97

    Google Scholar 

  407. Sakai, T.: How to run an evaluation task. In: Ferro and Peters [130]

    Google Scholar 

  408. Saleh, S., Pecina, P.: Reranking hypotheses of machine-translated queries for cross-lingual information retrieval. In: Fuhr et al. [143], pp. 54–68

    Google Scholar 

  409. Samuel, J.: Analyzing and visualizing translation patterns of Wikidata properties. In: Bellot et al. [42], pp. 128–134

    Google Scholar 

  410. Sánchez-Junquera, J., Villaseñor-Pineda, L., Montes-y Gómez, M., Rosso, P.: Character N-grams for detecting deceptive controversial opinions. In: Bellot et al. [42], pp. 135–140

    Google Scholar 

  411. Sanchez-Perez, M.A., Gelbukh, A.F., Sidorov, G.: Adaptive algorithm for plagiarism detection: the best-performing approach at PAN 2014 text alignment competition. In: Mothe et al. [301], pp. 402–413

    Google Scholar 

  412. Sanchez-Perez, M.A., Markov, I., Gómez-Adorno, H., Sidorov, G.: Comparison of character n-grams and lexical features on author, gender, and language variety identification on the same Spanish news corpus. In: Jones et al. [209], pp. 145–151

    Google Scholar 

  413. SanJuan, E., Moriceau, V., Tannier, X., Bellot, P., Mothe, J.: Overview of the INEX 2012 tweet contextualization track. In: Forner et al. [137]

    Google Scholar 

  414. Santos, D., Cabral, L.M.: GikiCLEF: expectations and lessons learned. In: Peters et al. [360], pp. 212–222

    Google Scholar 

  415. Ramos dos Santos, W., Paraboni, I.: Personality facets recognition from text. In: Crestani et al. [94]

    Google Scholar 

  416. Savenkov, D., Braslavski, P., Lebedev, M.: Search snippet evaluation at Yandex: lessons learned and future directions. In: Forner et al. [136], pp. 14–25

    Google Scholar 

  417. Savoy, J., Braschler, M.: Lessons learnt from experiments on the ad-hoc multilingual test collections at CLEF. In: Ferro and Peters [130]

    Google Scholar 

  418. Schaer, P.: Better than their reputation? On the reliability of relevance assessments with students. In: Catarci et al. [75], pp. 124–135

    Google Scholar 

  419. Schaer, P., Mayr, P., Sünkler, S., Lewandowski, D.: How relevant is the long tail?: a relevance assessment study on million short. In: Fuhr et al. [143], pp. 227–233

    Google Scholar 

  420. Schaer, P., Neumann, M.: Enriching existing test collections with OXPath. In: Jones et al. [209], pp. 152–158

    Google Scholar 

  421. Schaüble, P., Sheridan, P.: Cross-language information retrieval (CLIR) track overview. In: Voorhees, E.M., Harman, D.K. (eds.) The Sixth Text REtrieval Conference (TREC-6), pp. 31–44. National Institute of Standards and Technology (NIST), Special Publication 500-240, Washington, USA (1997)

    Google Scholar 

  422. Schubotz, M., Krämer, L., Meuschke, N., Hamborg, F., Gipp, B.: Evaluating and improving the extraction of mathematical identifier definitions. In: Jones et al. [209], pp. 82–94

    Google Scholar 

  423. Schuth, A., Balog, K., Kelly, L.: Overview of the living labs for information retrieval evaluation (LL4IR) CLEF Lab 2015. In: Mothe et al. [301], pp. 484–496

    Google Scholar 

  424. Schuth, A., Marx, M.: Evaluation methods for rankings of facetvalues for faceted search. In: Forner et al. [136], pp. 131–136

    Google Scholar 

  425. Shen, W., Nie, J.Y.: Is concept mapping useful for biomedical information retrieval. In: Mothe et al. [301], pp. 281–286

    Google Scholar 

  426. Shepeleva, N., Balog, K.: Towards an understanding of transactional tasks. In: Fuhr et al. [143], pp. 234–240

    Google Scholar 

  427. Sherkat, E., Velcin, J., Milios, E.E.: Fast and simple deterministic seeding of KMeans for text document clustering. In: Bellot et al. [42], pp. 76–88

    Google Scholar 

  428. Shing, H.S., Barrow, J., Galuščáková, P., Oard, D.W., Resnik, P.: Unsupervised system combination for set-based retrieval with expectation maximization. In: Crestani et al. [94]

    Google Scholar 

  429. Sierek, T., Hanbury, A.: Using health statistics to improve medical and health search. In: Mothe et al. [301], pp. 287–292

    Google Scholar 

  430. Sigurbjörnsson, B., Kamps, J., de Rijke, M.: Overview of WebCLEF 2005. In: Peters et al. [365], pp. 810–824

    Google Scholar 

  431. Silvello, G., Bordea, G., Ferro, N., Buitelaar, P., Bogers, T.: Semantic representation and enrichment of information retrieval experimental data. Int. J. Digit. Libr. (IJDL) 18(2), 145–172 (2017)

    Article  Google Scholar 

  432. Singh, G., Mantrach, A., Silvestri, F.: Improving profiles of weakly-engaged users: with applications to recommender systems. In: Fuhr et al. [143], pp. 123–136

    Google Scholar 

  433. Sorg, P., Cimiano, P., Schultz, A., Sizov, S.: Overview of the cross-lingual expert search (CriES) pilot challenge. In: Braschler et al. [61]

    Google Scholar 

  434. Spina, D., Amigó, E., Gonzalo, J.: Filter keywords and majority class strategies for company name disambiguation in Twitter. In: Forner et al. [136], pp. 38–49

    Google Scholar 

  435. Stamatatos, E., Potthast, M., Rangel Pardo, F.M., Rosso, P., Stein, B.: Overview of the PAN/CLEF 2015 evaluation lab. In: Mothe et al. [301], pp. 518–538

    Google Scholar 

  436. Stamatatos, E., et al.: Overview of PAN 2018. In: Bellot et al. [42], pp. 267–285

    Google Scholar 

  437. Stathopoulos, S., Kalamboukis, T.: Medical image classification with weighted latent semantic tensors and deep convolutional neural networks. In: Bellot et al. [42], pp. 89–100

    Google Scholar 

  438. Stefanov, V., Sachs, A., Kritz, M., Samwald, M., Gschwandtner, M., Hanbury, A.: A formative evaluation of a comprehensive search system for medical professionals. In: Forner et al. [138], pp. 81–92

    Google Scholar 

  439. Stempfhuber, M., Baerisch, S.: The domain-specific track at CLEF 2006: overview of approaches, results and assessment. In: Peters et al. [357], pp. 163–169

    Google Scholar 

  440. Suchomel, S., Brandejs, M.: Determining window size from plagiarism corpus for stylometric features. In: Mothe et al. [301], pp. 293–299

    Google Scholar 

  441. Suominen, H.: CLEFeHealth2012 - the CLEF 2012 workshop on cross-language evaluation of methods, applications, and resources for eHealth document analysis. In: Forner et al. [137]

    Google Scholar 

  442. Suominen, H., Kelly, L., Goeuriot, L.: The scholarly impact and strategic intent of CLEF eHealth labs from 2012–2017. In: Ferro and Peters [130]

    Google Scholar 

  443. Suominen, H., et al.: Overview of the CLEF eHealth evaluation lab 2018. In: Bellot et al. [42], pp. 286–301

    Google Scholar 

  444. Suominen, H., et al.: Overview of the CLEF eHealth evaluation lab 2019. In: Crestani et al. [94]

    Google Scholar 

  445. Suominen, H., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner et al. [138], pp. 212–231

    Google Scholar 

  446. Tannebaum, W., Mahdabi, P., Rauber, A.: Effect of log-based query term expansion on retrieval effectiveness in patent searching. In: Mothe et al. [301], pp. 300–305

    Google Scholar 

  447. Tannebaum, W., Rauber, A.: Mining query logs of USPTO patent examiners. In: Forner et al. [138], pp. 136–142

    Google Scholar 

  448. Teixeira Lopes, C., Almeida Fernandes, T.: Health suggestions: a chrome extension to help laypersons search for health information. In: Fuhr et al. [143], pp. 241–246

    Google Scholar 

  449. Teixeira Lopes, C., Ribeiro, C.: Effects of language and terminology on the usage of health query suggestions. In: Fuhr et al. [143], pp. 83–95

    Google Scholar 

  450. Teixeira Lopes, C., Ribeiro, C.: Effects of language and terminology of query suggestions on the precision of health searches. In: Bellot et al. [42], pp. 101–111

    Google Scholar 

  451. Thomee, B., Popescu, A.: Overview of the ImageCLEF 2012 Flickr photo annotation and retrieval task. In: Forner et al. [137]

    Google Scholar 

  452. Thornley, C.V., Johnson, A.C., Smeaton, A.F., Lee, H.: The scholarly impact of TRECVid (2003–2009). J. Am. Soc. Inf. Sci. Technol. (JASIST) 62(4), 613–627 (2011)

    Article  Google Scholar 

  453. Tommasi, T., Caputo, B., Welter, P., Güld, M.O., Deserno, T.M.: Overview of the CLEF 2009 medical image annotation track. In: Peters et al. [368], pp. 85–93

    Google Scholar 

  454. Trappett, M., Geva, S., Trotman, A., Scholer, F., Sanderson, M.: Overview of the INEX 2012 snippet retrieval track. In: Forner et al. [137]

    Google Scholar 

  455. Trotzek, M., Koitka, S., Friedrich, C.M.: Early detection of depression based on linguistic metadata augmented classifiers revisited. In: Bellot et al. [42], pp. 191–202

    Google Scholar 

  456. Tsikrika, T., Garcia Seco de Herrera, A., Müller, H.: Assessing the scholarly impact of ImageCLEF. In: Forner et al. [136], pp. 95–106

    Google Scholar 

  457. Tsikrika, T., Kludas, J.: Overview of the WikipediaMM task at ImageCLEF 2008. In: Peters et al. [359], pp. 539–550

    Google Scholar 

  458. Tsikrika, T., Kludas, J.: Overview of the WikipediaMM task at ImageCLEF 2009. In: Peters et al. [368], pp. 60–71

    Google Scholar 

  459. Tsikrika, T., Larsen, B., Müller, H., Endrullis, S., Rahm, E.: The scholarly impact of CLEF (2000–2009). In: Forner et al. [138], pp. 1–12

    Google Scholar 

  460. Tsikrika, T., Popescu, A., Kludas, J.: Overview of the Wikipedia image retrieval task at ImageCLEF 2011. In: Petras et al. [373]

    Google Scholar 

  461. Turchi, M., Steinberger, J., Alexandrov Kabadjov, M., Steinberger, R.: Using parallel corpora for multilingual (multi-document) summarisation evaluation. In: Agosti et al. [9], pp. 52–63

    Google Scholar 

  462. Turmo, J., et al.: Overview of QAST 2009. In: Peters et al. [360], pp. 197–211

    Google Scholar 

  463. Turmo, J., Comas, P., Rosset, S., Lamel, L., Moreau, N., Mostefa, D.: Overview of QAST 2008. In: Peters et al. [359], pp. 296–313

    Google Scholar 

  464. Vallin, A., et al.: Overview of the CLEF 2005 multilingual question answering track. In: Peters et al. [365], pp. 307–331

    Google Scholar 

  465. Valverde-Albacete, F., Carrillo de Albornoz, J., Peláez-Moreno, C.: A proposal for new evaluation metrics and result visualization technique for sentiment analysis tasks. In: Forner et al. [138], pp. 41–42

    Google Scholar 

  466. Villegas, M., et al.: General overview of ImageCLEF at the CLEF 2015 labs. In: Mothe et al. [301], pp. 444–461

    Google Scholar 

  467. Villegas, M., et al.: General overview of ImageCLEF at the CLEF 2016 labs. In: Fuhr et al. [143], pp. 267–285

    Google Scholar 

  468. Villegas, M., Paredes, R.: Overview of the ImageCLEF 2012 scalable web image annotation task. In: Forner et al. [137]

    Google Scholar 

  469. Voorhees, E.M.: TREC: continuing information retrieval’s tradition of experimentation. Commun. ACM (CACM) 50(11), 51–54 (2007)

    Article  Google Scholar 

  470. Voorhees, E.M.: The evolution of cranfield. In: Ferro and Peters [130]

    Google Scholar 

  471. Wakeling, S., Clough, P.: Integrating mixed-methods for evaluating information access systems. In: Mothe et al. [301], pp. 306–311

    Google Scholar 

  472. Walker, A., Starkey, A., Pan, J.Z., Siddharthan, A.: Making test corpora for question answering more representative. In: Kanoulas et al. [216], pp. 1–6

    Google Scholar 

  473. Wang, J., Gilbert, A., Thomee, B., Villegas, M.: Automatic image annotation at ImageCLEF. In: Ferro and Peters [130]

    Google Scholar 

  474. Wang, Q., et al.: Overview of the INEX 2012 linked data track. In: Forner et al. [137]

    Google Scholar 

  475. Wang, X., Guo, Z., Zhang, Y., Li, J.: Medical image labeling and semantic understanding for clinical applications. In: Crestani et al. [94]

    Google Scholar 

  476. Wang, X., Wang, X., Zhang, Q.: A web-based CLIR system with cross-lingual topical pseudo relevance feedback. In: Forner et al. [138], pp. 104–107

    Google Scholar 

  477. Weitzel, L., Bernardini, F., Quaresma, P., Alves, C.A., Zacharski, W., de Figueiredo, L.G.: Brazilian social mood: the political dimension of emotion. In: Fuhr et al. [143], pp. 247–255

    Google Scholar 

  478. White, R.W., Oard, D.W., Jones, G.J.F., Soergel, D., Huang, X.: Overview of the CLEF-2005 cross-language speech retrieval track. In: Peters et al. [365], pp. 744–759

    Google Scholar 

  479. Wilhelm-Stein, T., Eibl, M.: A quantitative look at the CLEF working notes. In: Forner et al. [138], pp. 13–16

    Google Scholar 

  480. Wilhelm-Stein, T., Eibl, M.: Teaching the IR process using real experiments supported by game mechanics. In: Mothe et al. [301], pp. 312–317

    Google Scholar 

  481. Wilhelm-Stein, T., Herms, R., Ritter, M., Eibl, M.: Improving transcript-based video retrieval using unsupervised language model adaptation. In: Kanoulas et al. [216], pp. 110–115

    Google Scholar 

  482. Xu, K., Feng, Y., Huang, S., Zhao, D.: Question answering via phrasal semantic parsing. In: Mothe et al. [301], pp. 414–426

    Google Scholar 

  483. Yan, X., Gao, G., Su, X., Wei, H., Zhang, X., Lu, Q.: Hidden Markov model for term weighting in verbose queries. In: Catarci et al. [75], pp. 82–87

    Google Scholar 

  484. Yang, H., Gonçalves, T.: A compound model for consumer health search. In: Bellot et al. [42], pp. 231–236

    Google Scholar 

  485. Zamani, H., Esfahani, H.N., Babaie, P., Abnar, S., Dehghani, M., Shakery, A.: Authorship identification using dynamic selection of features from probabilistic feature set. In: Kanoulas et al. [216], pp. 128–140

    Google Scholar 

  486. Zellhöfer, D.: Overview of the personal photo retrieval pilot task at ImageCLEF 2012. In: Forner et al. [137]

    Google Scholar 

  487. Zhang, L., Rettinger, A., Färber, M., Tadic, M.: A comparative evaluation of cross-lingual text annotation techniques. In: Forner et al. [138], pp. 124–135

    Google Scholar 

  488. Ziak, H., Kern, R.: Evaluation of pseudo relevance feedback techniques for cross vertical aggregated search. In: Mothe et al. [301], pp. 91–102

    Google Scholar 

  489. Zingla, M.A., Latiri, C., Slimani, Y.: Tweet contextualization using association rules mining and DBpedia. In: Mothe et al. [301], pp. 318–323

    Google Scholar 

  490. Zlabinger, M., Rekabsaz, N., Zlabinger, S., Hanbury, A.: Efficient answer-annotation for frequent questions. In: Crestani et al. [94]

    Google Scholar 

  491. Zuo, C., Karakas, A., Banerjee, R.: To check or not to check: syntax, semantics, and context in the language of check-worthy claims. In: Crestani et al. [94]

    Google Scholar 

Download references

Acknowledgements

CLEF would not be possible without all the effort, enthusiasm, and passion of its community: lab organizers, lab participants, and attendees are the core and the real success of CLEF.

Many friends and colleagues – too many to mention them all but I sincerely thank all of them – have shared with me this journey through CLEF and their work, passion, ideas, expertise and wisdom have shaped what CLEF is today.

However, all of this would have not even been possible without Carol Peters, who established CLEF back in 2000, made it grown over the years with constant care, and put into CLEF her secret ingredient which makes it so special: a very friendly environment where everybody feels to be welcome and comfortable in sharing ideas and contributions. Carol has had the generosity of sharing her experience with me and teaching me a lot about how to run an evaluation initiative and grow a healthy research community: I will never thank her enough for this.

Author information

Authors and Affiliations

  1. Department of Information Engineering, University of Padua, Padua, Italy

    Nicola Ferro

Authors
  1. Nicola Ferro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicola Ferro .

Editor information

Editors and Affiliations

  1. Universita della Svizzera Italiana, Lugano, Switzerland

    Fabio Crestani

  2. Zurich University of Applied Sciences, Winterthur, Switzerland

    Martin Braschler

  3. University of Neuchâtel, Neuchâtel, Switzerland

    Jacques Savoy

  4. Technische Universität Wien, Vienna, Austria

    Andreas Rauber

  5. HES-SO Valais-Wallis, Sierre, Switzerland

    Henning Müller

  6. University of Santiago de Compostela, Santiago de Compostela, Spain

    David E. Losada

  7. Swiss Alliance for Data-Intensive Services, Thun, Switzerland

    Gundula Heinatz Bürki

  8. University of Padua, Padua, Italy

    Linda Cappellato

  9. University of Padua, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ferro, N. (2019). What Happened in CLEF\(\ldots \) For a While?. In: Crestani, F., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2019. Lecture Notes in Computer Science(), vol 11696. Springer, Cham. https://doi.org/10.1007/978-3-030-28577-7_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-28577-7_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-28576-0

  • Online ISBN: 978-3-030-28577-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation