Skip to main content
Book cover

International Conference of the Cross-Language Evaluation Forum for European Languages

CLEF 2017: Experimental IR Meets Multilinguality, Multimodality, and Interaction pp 361–366Cite as

CLEF 2017 Dynamic Search Evaluation Lab Overview

Part of the Lecture Notes in Computer Science book series (LNISA,volume 10456)

Abstract

In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-65813-1_31
  • Chapter length: 6 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-65813-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)
Learn about institutional subscriptions

References

  1. Allan, J.: Hard track overview in TREC 2003 high accuracy retrieval from documents. Technical report, DTIC Document (2005)

    Google Scholar 

  2. Balog, K.: Task-completion engines: a vision with a plan. In: SCST@ECIR (2015)

    Google Scholar 

  3. Carterette, B., Clough, P.D., Hall, M.M., Kanoulas, E., Sanderson, M.: Evaluating retrieval over sessions: the TREC session track 2011–2014. In: Perego, R., Sebastiani, F., Aslam, J.A., Ruthven, I., Zobel, J. (eds.) Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, 17–21 July 2016, pp. 685–688. ACM (2016). http://doi.acm.org/10.1145/2911451.2914675

  4. Georgila, K., Henderson, J., Lemon, O.: User simulation for spoken dialogue systems: learning and evaluation. In: Interspeech pp. 1065–1068 (2006)

    Google Scholar 

  5. Jung, S., Lee, C., Kim, K., Jeong, M., Lee, G.G.: Data-driven user simulation for automated evaluation of spoken dialog systems. Comput. Speech Lang. 23(4), 479–509 (2009). http://dx.doi.org/10.1016/j.csl.2009.03.002

    CrossRef  Google Scholar 

  6. Maxwell, D., Azzopardi, L.: Agents, simulated users and humans: An analysis of performance and behaviour. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM 2016, pp. 731–740 (2016)

    Google Scholar 

  7. Over, P.: The TREC interactive track: an annotated bibliography. Inf. Process. Manage. 37(3), 369–381 (2001)

    CrossRef  MATH  Google Scholar 

  8. Pääkkönen, T., Kekäläinen, J., Keskustalo, H., Azzopardi, L., Maxwell, D., Järvelin, K.: Validating simulated interaction for retrieval evaluation. Inf. Retrieval J., 1–25 (2017)

    Google Scholar 

  9. Pietquin, O., Hastie, H.: A survey on metrics for the evaluation of user simulations. Knowl. Eng. Rev. 28(01), 59–73 (2013)

    CrossRef  Google Scholar 

  10. Serban, I.V., Lowe, R., Henderson, P., Charlin, L., Pineau, J.: A survey of available corpora for building data-driven dialogue systems. CoRR abs/1512.05742 (2015). http://arxiv.org/abs/1512.05742

  11. Verma, M., Yilmaz, E., Mehrotra, R., Kanoulas, E., Carterette, B., Craswell, N., Bailey, P.: Overview of the TREC tasks track 2016. In: Voorhees, E.M., Ellis, A. (eds.) Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, 15–18 November 2016, vol. Special Publication 500–321. National Institute of Standards and Technology (NIST) (2016). http://trec.nist.gov/pubs/trec25/papers/Overview-T.pdf

  12. Yilmaz, E., Verma, M., Mehrotra, R., Kanoulas, E., Carterette, B., Craswell, N.: Overview of the TREC 2015 tasks track. In: Voorhees, E.M., Ellis, A. (eds.) Proceedings of The Twenty-Fourth Text REtrieval Conference, TREC 2015, Gaithersburg, Maryland, USA, 17–20 November 2015, vol. Special Publication 500–319. National Institute of Standards and Technology (NIST) (2015). http://trec.nist.gov/pubs/trec24/papers/Overview-T.pdf

Download references

Acknowledgements

This work was partially supported by the Google Faculty Research Award program and the Microsoft Azure for Research Award program (CRM:0518163). All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors. We would also like to thank Dr. Guido Zuccon for setting up the ElasticSearch API.

Author information

Authors and Affiliations

  1. Informatics Institute, University of Amsterdam, Amsterdam, Netherlands

    Evangelos Kanoulas

  2. Computer and Information Sciences, University of Strathclyde, Glasgow, UK

    Leif Azzopardi

Authors
  1. Evangelos Kanoulas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leif Azzopardi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Evangelos Kanoulas .

Editor information

Editors and Affiliations

  1. Dublin City University, Dublin, Ireland

    Gareth J.F. Jones

  2. Trinity College Dublin, Dublin, Ireland

    Séamus Lawless

  3. National University of Distance Education, Madrid, Spain

    Julio Gonzalo

  4. Dublin City University, Dublin, Ireland

    Liadh Kelly

  5. Université Grenoble Alpes, Grenoble, France

    Lorraine Goeuriot

  6. University of Hildesheim, Hildesheim, Germany

    Thomas Mandl

  7. University of Padua, Padua, Italy

    Linda Cappellato

  8. University of Padua, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and Permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Kanoulas, E., Azzopardi, L. (2017). CLEF 2017 Dynamic Search Evaluation Lab Overview. In: , et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2017. Lecture Notes in Computer Science(), vol 10456. Springer, Cham. https://doi.org/10.1007/978-3-319-65813-1_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-65813-1_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-65812-4

  • Online ISBN: 978-3-319-65813-1

  • eBook Packages: Computer ScienceComputer Science (R0)