Information Retrieval Evaluation in a Changing World

Lessons Learned from 20 Years of CLEF

  • Nicola Ferro
  • Carol Peters

Part of the The Information Retrieval Series book series (INRE, volume 41)

Table of contents

  1. Front Matter
    Pages i-xxii
  2. Experimental Evaluation and CLEF

    1. Front Matter
      Pages 1-1
    2. Ellen M. Voorhees
      Pages 45-69
    3. Tetsuya Sakai
      Pages 71-102
  3. Evaluation Infrastructures

    1. Front Matter
      Pages 103-103
    2. Maristella Agosti, Giorgio Maria Di Nunzio, Nicola Ferro, Gianmaria Silvello
      Pages 105-122
    3. Martin Potthast, Tim Gollub, Matti Wiegmann, Benno Stein
      Pages 123-160
  4. Multilingual and Multimedia Information Retrieval

    1. Front Matter
      Pages 175-175
    2. Jussi Karlgren, Turid Hedlund, Kalervo Järvelin, Heikki Keskustalo, Kimmo Kettunen
      Pages 201-216
    3. Paul Clough, Theodora Tsikrika
      Pages 217-230
    4. Henning Müller, Jayashree Kalpathy-Cramer, Alba García Seco de Herrera
      Pages 231-250
    5. Josiah Wang, Andrew Gilbert, Bart Thomee, Mauricio Villegas
      Pages 251-273
    6. Luca Piras, Barbara Caputo, Duc-Tien Dang-Nguyen, Michael Riegler, Pål Halvorsen
      Pages 275-305
  5. Retrieval in New Domains

    1. Front Matter
      Pages 331-331
    2. Hanna Suominen, Liadh Kelly, Lorraine Goeuriot
      Pages 333-363
    3. Florina Piroi, Allan Hanbury
      Pages 365-387
    4. Alexis Joly, Hervé Goëau, Hervé Glotin, Concetto Spampinato, Pierre Bonnet, Willem-Pier Vellinga et al.
      Pages 389-413
    5. Jaap Kamps, Marijn Koolen, Shlomo Geva, Ralf Schenkel, Eric SanJuan, Toine Bogers
      Pages 415-437
  6. Beyond Retrieval

    1. Front Matter
      Pages 439-439
    2. Anselmo Peñas, Álvaro Rodrigo, Bernardo Magnini, Pamela Forner, Eduard Hovy, Richard Sutcliffe et al.
      Pages 441-460
    3. Paolo Rosso, Martin Potthast, Benno Stein, Efstathios Stamatatos, Francisco Rangel, Walter Daelemans
      Pages 461-485
    4. Jorge Carrillo-de-Albornoz, Julio Gonzalo, Enrique Amigó
      Pages 487-510
    5. Frank Hopfgartner, Krisztian Balog, Andreas Lommatzsch, Liadh Kelly, Benjamin Kille, Anne Schuth et al.
      Pages 511-543
  7. Impact and Future Challenges

    1. Front Matter
      Pages 545-545
    2. Birger Larsen
      Pages 547-554
    3. Norbert Fuhr
      Pages 555-564
    4. Nicola Ferro, Giuseppe Santucci
      Pages 565-582
  8. Back Matter
    Pages 591-595

About this book


This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since – and traces its evolution over these first two decades. CLEF’s main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation.

The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing “what has been achieved”, but above all on “what has been learnt”. The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings.

Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval.


Information Retrieval Multilingual Retrieval Multimedia Retrieval Natural Language Processing Patent Retrieval Visual Analytics

Editors and affiliations

  1. 1.Dipartimento di Ingegneria dell’InformazioneUniversità degli Studi di Padova PadovaItaly
  2. 2.Consiglio Nazionale delle RicercheIstituto di Scienza e Tecnologie dell’InformazionePisaItaly

Bibliographic information