CLEF 2017 eHealth Evaluation Lab Overview

  • Lorraine Goeuriot
  • Liadh Kelly
  • Hanna SuominenEmail author
  • Aurélie Névéol
  • Aude Robert
  • Evangelos Kanoulas
  • Rene Spijker
  • João Palotti
  • Guido Zuccon
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10456)


In this paper we provide an overview of the fifth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2017 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on multilingual information extraction to extend from last year’s task on French corpora, Task 2 on technologically assisted reviews in empirical medicine as a new pilot task, and Task 3 on patient-centered information retrieval (IR) building on the 2013-16 IR tasks. In total 32 teams took part in these tasks (11 in Task 1, 14 in Task 2, and 7 in Task 3). We also continued the replication track from 2016. Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants of this year’s challenges and results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.


Evaluation Entity linking Information retrieval Health records Information extraction Medical informatics Systematic reviews Test-set generation Text classification Text segmentation Self-diagnosis 



The CLEF eHealth 2017 evaluation lab has been supported in part by (in alphabetical order) the ANR, the French National Research Agency, under grant CABeRneT ANR-13-JS02-0009-01, the CLEF Initiative and Data61. We are also thankful to the people involved in the annotation, query creation, and relevance assessment exercise. Last but not least, we gratefully acknowledge the participating teams’ hard work. We thank them for their submissions and interest in the lab.


  1. 1.
    Cormack, G.V., Grossman, M.R.: Engineering quality and reliability in technology-assisted review. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016, pp. 75–84. ACM, New York (2016).
  2. 2.
    Cormack, G.V., Smucker, M.D., Clarke, C.L.: Efficient and effective spam filtering and re-ranking for large web datasets. Inf. Retrieval 14(5), 441–465 (2011)CrossRefGoogle Scholar
  3. 3.
    Goeuriot, L., Jones, G.J., Kelly, L., Leveling, J., Hanbury, A., Müller, H., Salantera, S., Suominen, H., Zuccon, G.: ShARe, CLEF eHealth Evaluation Lab 2013, Task 3: Information retrieval to address patients’ questions when reading clinical reports. CLEF 2013 Online Working. Notes 8138 (2013)Google Scholar
  4. 4.
    Goeuriot, L., Kelly, L., Lee, W., Palotti, J., Pecina, P., Zuccon, G., Hanbury, A., Gareth J.F. Jones, H.M.: ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: user-centred health information retrieval. In: CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield, UK (2014)Google Scholar
  5. 5.
    Goeuriot, L., Kelly, L., Suominen, H., Hanlen, L., Névéol, A., Grouin, C., Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2015. In: Mothe, J., Savoy, J., Kamps, J., Pinel-Sauvagnat, K., Jones, G.J.F., SanJuan, E., Cappellato, L., Ferro, N. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 429–443. Springer, Cham (2015). doi: 10.1007/978-3-319-24027-5_44 CrossRefGoogle Scholar
  6. 6.
    Hanbury, A., Müller, H.: Khresmoi - multimodal multilingual medical information search. In: Medical Informatics Europe 2012 (MIE 2012), Village of the Future (2012)Google Scholar
  7. 7.
    Hiemstra, D., Hauff, C.: Mirex: Mapreduce information retrieval experiments. arXiv preprint arXiv:1004.4489
  8. 8.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)CrossRefGoogle Scholar
  9. 9.
    Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: Overview of the CLEF technologically assisted reviews in empirical medicine. In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings (2017)Google Scholar
  10. 10.
    Kelly, L., Goeuriot, L., Suominen, H., Névéol, A., Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2016. In: Fuhr, N., Quaresma, P., Gonçalves, T., Larsen, B., Balog, K., Macdonald, C., Cappellato, L., Ferro, N. (eds.) CLEF 2016. LNCS, vol. 9822, pp. 255–266. Springer, Cham (2016). doi: 10.1007/978-3-319-44564-9_24 CrossRefGoogle Scholar
  11. 11.
    Kelly, L., Goeuriot, L., Suominen, H., Schreck, T., Leroy, G., Mowery, D.L., Velupillai, S., Chapman, W.W., Martinez, D., Zuccon, G., Palotti, J.: Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 172–191. Springer, Cham (2014). doi: 10.1007/978-3-319-11382-1_17 Google Scholar
  12. 12.
    Leeflang, M.M., Deeks, J.J., Takwoingi, Y., Macaskill, P.: Cochrane diagnostic test accuracy reviews. Syst. Rev. 2(1), 82 (2013)CrossRefGoogle Scholar
  13. 13.
    Macdonald, C., McCreadie, R., Santos, R.L., Ounis, I.: From puppy to maturity: experiences in developing terrier. In: Proceeding of OSIR at SIGIR, pp. 60–63 (2012)Google Scholar
  14. 14.
    Névéol, A., Anderson, R.N., Cohen, K.B., Grouin, C., Lavergne, T., Rey, G., Robert, A., Zweigenbaum, P.: CLEF eHealth 2017 multilingual information extraction task overview: ICD10 coding of death certificates in English and French. In: CLEF 2017 Online Working Notes. CEUR-WS (2017)Google Scholar
  15. 15.
    Névéol, A., Cohen, K.B., Grouin, C., Hamon, T., Lavergne, T., Kelly, L., Goeuriot, L., Rey, G., Robert, A., Tannier, X., Zweigenbaum, P.: Clinical information extraction at the CLEF eHealth evaluation lab 2016. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS (2016)Google Scholar
  16. 16.
    Palotti, J., Zuccon, G., Goeuriot, L., Kelly, L., Hanburyn, A., Jones, G.J., Lupu, M., Pecina, P.: CLEF eHealth evaluation lab 2015, Task 2: retrieving information about medical symptoms. In: CLEF 2015 Online Working Notes. CEUR-WS (2015)Google Scholar
  17. 17.
    Palotti, J., Zuccon, G., Jimmy, Pecina, P., Lupu, M., Goeuriot, L., Kelly, L., Hanbury, A.: CLEF 2017 task overview: the IR task at the eHealth evaluation lab. In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings (2017)Google Scholar
  18. 18.
    Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis, vol. 2, pp. 2–6. Citeseer (2005)Google Scholar
  19. 19.
    Suominen, H., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 212–231. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40802-1_24 CrossRefGoogle Scholar
  20. 20.
    Zuccon, G.: Understandability biased evaluation for information retrieval. In: Advances in Information Retrieval, pp. 280–292 (2016)Google Scholar
  21. 21.
    Zuccon, G., Koopman, B.: Integrating understandability in the evaluation of consumer health search engines. In: Medical Information Retrieval Workshop at SIGIR 2014, p. 32 (2014)Google Scholar
  22. 22.
    Zuccon, G., Palotti, J., Goeuriot, L., Kelly, L., Lupu, M., Pecina, P., Mueller, H., Budaher, J., Deacon, A.: The IR task at the CLEF eHealth evaluation lab 2016: user-centred health information retrieval. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS, September 2016Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Lorraine Goeuriot
    • 1
  • Liadh Kelly
    • 2
  • Hanna Suominen
    • 3
    • 4
    • 5
    Email author
  • Aurélie Névéol
    • 6
  • Aude Robert
    • 7
  • Evangelos Kanoulas
    • 8
  • Rene Spijker
    • 9
  • João Palotti
    • 10
  • Guido Zuccon
    • 11
  1. 1.University Grenoble Alpes, CNRS, Grenoble INP, LIGGrenobleFrance
  2. 2.ADAPT CentreDublin City UniversityDublinIreland
  3. 3.Data61/CSIROThe Australian National University (ANU)CanberraAustralia
  4. 4.University of CanberraCanberraAustralia
  5. 5.University of TurkuTurkuFinland
  6. 6.LIMSI CNRS UPR 3251 Université Paris-SaclayOrsayFrance
  7. 7.INSERM - CépiDcLe Kremlin-Bicêtre CedexFrance
  8. 8.Informatics InstituteUniversity of AmsterdamAmsterdamNetherlands
  9. 9.Julius Center for Health Sciences and Primary CareCochrane Netherlands and UMC UtrechtUtrechtNetherlands
  10. 10.Vienna University of TechnologyViennaAustria
  11. 11.Queensland University of TechnologyBrisbaneAustralia

Personalised recommendations