Skip to main content
SpringerLink
Log in

Overview of the CLEF eHealth Evaluation Lab 2015

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2015)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9283))

Abstract

This paper reports on the 3rd CLEFeHealth evaluation lab, which continues our evaluation resource building activities for the medical domain. In this edition of the lab, we focus on easing patients and nurses in authoring, understanding, and accessing eHealth information. The 2015 CLEFeHealth evaluation lab was structured into two tasks, focusing on evaluating methods for information extraction (IE) and information retrieval (IR). The IE task introduced two new challenges. Task 1a focused on clinical speech recognition of nursing handover notes; Task 1b focused on clinical named entity recognition in languages other than English, specifically French. Task 2 focused on the retrieval of health information to answer queries issued by general consumers seeking information to understand their health symptoms or conditions.

The number of teams registering their interest was 47 in Tasks 1 (2 teams in Task 1a and 7 teams in Task 1b) and 53 in Task 2 (12 teams) for a total of 20 unique teams. The best system recognized 4, 984 out of 6, 818 test words correctly and generated 2, 626 incorrect words (i.e., \(38.5 \%\) error) in Task 1a; had the F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition, and 0.872 for entity normalization in Task 1b; and resulted in P@10 of 0.5394 and nDCG@10 of 0.5086 in Task 2. These results demonstrate the substantial community interest and capabilities of these systems in addressing challenges faced by patients and nurses. As in previous years, the organizers have made data and tools available for future research and development.

G. Zuccon—In alphabetical order, LG & LK co-chaired the lab. In order of contribution, HS & LH led Task 1a. In order of contribution, AN & CG led Task 1B. In alphabetical order, JP & GZ led Task 2.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Suominen, H., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 212–231. Springer, Heidelberg (2013)

    Google Scholar 

  2. Pradhan, S., Elhadad, N., South, B., Martinez, D., Christensen, L., Vogel, A., Suominen, H., Chapman, W., Savova, G.: Task 1: ShARe/CLEF eHealth evaluation lab 2013. In: Online Working Notes of CLEF, CLEF (2013)

    Google Scholar 

  3. Mowery, D., South, B., Christensen, L., Murtola, L., Salanterä, S., Suominen, H., Martinez, D., Elhadad, N., Pradhan, S., Savova, G., Chapman, W.: Task 2: ShARe/CLEF eHealth Evaluation Lab 2013. In: Online Working Notes of CLEF, CLEF (2013)

    Google Scholar 

  4. Goeuriot, L., Jones, G., Kelly, L., Leveling, J., Hanbury, A., Müller, H., Salanterä, S., Suominen, H., Zuccon, G.: ShARe/CLEF eHealth evaluation lab 2013, task 3: information retrieval to address patients’ questions when reading clinical reports. In: Online Working Notes of CLEF, CLEF (2013)

    Google Scholar 

  5. Kelly, L., Goeuriot, L., Schreck, T., Leroy, G., Mowery, D.L., Velupillai, S., Chapman, W., Martinez, D., Zuccon, G., Palotti, J.: Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 172–191. Springer, Heidelberg (2014)

    Google Scholar 

  6. Suominen, H., Schreck, T., Leroy, G., Hochheiser, H., Goeuriot, L., Kelly, L., Mowery, D., Nualart, J., Ferraro, G., Keim, D.: Task 1 of the CLEF eHealth Evaluation Lab 2014: visual-interactive search and exploration of eHealth data. In: CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield, UK (2014)

    Google Scholar 

  7. Mowery, D., Velupillai, S., South, B., Christensen, L., Martinez, D., Kelly, L., Goeuriot, L., Elhadad, N., Pradhan, S., Savova, G., Chapman, W.: Task 2 of the CLEF eHealth Evaluation Lab 2014: Information extraction from clinical text. In: CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield, UK (2014)

    Google Scholar 

  8. Goeuriot, L., Kelly, L., Lee, W., Palotti, J., Pecina, P., Zuccon, G., Hanbury, A., Gareth, J.F., Jones, H.M.: ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred health information retrieval. In: CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield, UK (2014)

    Google Scholar 

  9. Suominen, H., Hanlen, L., Goeuriot, L., Kelly, L., Jones, G.J.: Task 1a of the CLEF eHealth evaluation lab 2015: Clinical speech recognition. In: CLEF 2015 Online Working Notes, CEUR-WS (2015)

    Google Scholar 

  10. Névéol, A., Grouin, C., Tannier, X., Hamon, T., Kelly, L., Goeuriot, L., Zweigenbaum, P.: CLEF eHealth evaluation lab 2015 task 1b: clinical named entity recognition. In: CLEF 2015 Online Working Notes, CEUR-WS (2015)

    Google Scholar 

  11. Palotti, J., Zuccon, G., Goeuriot, L., Kelly, L., Hanburyn, A., Jones, G.J., Lupu, M., Pecina, P.: CLEF eHealth evaluation lab 2015, task 2: Retrieving information about medical symptoms. In: CLEF 2015 Online Working Notes, CEUR-WS (2015)

    Google Scholar 

  12. Névéol, A., Dalianis, H., Savova, G., Zweigenbaum, P.: Didactic panel: clinical natural language processing in languages other than English. In: Proc AMIA Annu. Symp. (2014)

    Google Scholar 

  13. Goeuriot, L., Kelly, L., Jones, G.J., Zuccon, G., Suominen, H., Hanbury, A., Mueller, H., Leveling, J.: Creation of a new evaluation benchmark for information retrieval targeting patient information needs. In: The Fifth International Workshop on Evaluating Information Access (EVIA 2013), vol. 18 (2013)

    Google Scholar 

  14. Fox, S.: Health topics: 80% of internet users look for health information online. Pew Internet & American Life Project (2011)

    Google Scholar 

  15. Benigeri, M., Pluye, P.: Shortcomings of health information on the internet. Health Promotion International 18(4), 381–386 (2003)

    Article  Google Scholar 

  16. White, R.W., Horvitz, E.: Cyberchondria: studies of the escalation of medical concerns in web search. ACM TOIS 27(4), 23 (2009)

    Article  Google Scholar 

  17. Zuccon, G., Koopman, B., Palotti, J.: Diagnose this if you can. In: Hanbury, A., Kazai, G., Rauber, A., Fuhr, N. (eds.) ECIR 2015. LNCS, vol. 9022, pp. 562–567. Springer, Heidelberg (2015)

    Google Scholar 

  18. Suominen, H., Zhou, L., Hanlen, L., Ferraro, G.: Benchmarking clinical speech recognition and information extraction: New data, methods and evaluations. JMIR Medical Informatics 3(2), e19 (2015)

    Article  Google Scholar 

  19. Hanbury, A., Müller, H.: Khresmoi - multimodal multilingual medical information search. In: MIE Village of the Future (2012)

    Google Scholar 

  20. Bodenreider, O., McCray, A.T.: Exploring semantic groups through visual approaches. J. Biomed. Inform. 36(6), 414–432 (2003)

    Article  Google Scholar 

  21. Névéol, A., Grouin, C., Leixa, J., Rosset, S., Zweigenbaum, P.: The QUAERO French medical corpus: A resource for medical entity recognition and normalization. In: Proc. of BioTextMining Work., pp. 24–30 (2014)

    Google Scholar 

  22. Stanton, I., Ieong, S., Mishra, N.: Circumlocution in diagnostic medical queries. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 133–142. ACM (2014)

    Google Scholar 

  23. Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bulletin 1(6), 80–83 (1945)

    Article  Google Scholar 

  24. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20(4), 422–446 (2002)

    Article  Google Scholar 

  25. Zuccon, G., Koopman, B.: Integrating understandability in the evaluation of consumer health search engines. In: Medical Information Retrieval Workshop at SIGIR 2014, p. 32 (2014)

    Google Scholar 

  26. Verspoor, K., Yepes, A.J., Cavedon, L., McIntosh, T., Herten-Crabb, A., Thomas, Z., Plazzer, J.P.: Annotating the biomedical literature for the human variome. Database (Oxford), bat019-bat019 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LIG, Université Grenoble Alpes, Grenoble, France

    Lorraine Goeuriot

  2. ADAPT Centre, Trinity College, Dublin, Ireland

    Liadh Kelly

  3. NICTA, Canberra, ACT, Australia

    Hanna Suominen & Leif Hanlen

  4. The Australian National University, Canberra, ACT, Australia

    Hanna Suominen & Leif Hanlen

  5. University of Canberra, Canberra, ACT, Australia

    Hanna Suominen & Leif Hanlen

  6. University of Turku, Turku, Finland

    Hanna Suominen

  7. LIMSI CNRS UPR 3251, Orsay, France

    Aurélie Névéol & Cyril Grouin

  8. Vienna University of Technology, Vienna, Austria

    João Palotti

  9. Queensland University of Technology, Brisbane, Australia

    Guido Zuccon

Authors
  1. Lorraine Goeuriot
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liadh Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanna Suominen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leif Hanlen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aurélie Névéol
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Cyril Grouin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. João Palotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Guido Zuccon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lorraine Goeuriot .

Editor information

Editors and Affiliations

  1. Institut de Recherche en Informatique de Toulouse, Toulouse , France

    Josanne Mothe

  2. Department of Computer Science, University of Neuchatel, Neuchâtel, Switzerland

    Jacques Savoy

  3. Faculteit der Geesteswetenschappen, Universiteit Amsterdam, Amsterdam, The Netherlands

    Jaap Kamps

  4. Institut de Recherche en Informatique de Toulouse, Toulouse, France

    Karen Pinel-Sauvagnat

  5. School of Computing, Dublin City University, Dublin, Ireland

    Gareth Jones

  6. LIA - CERI, Université d'Avignon et des Pays de Vaucluse, Avignon, France

    Eric San Juan

  7. Department of Information Engineering, University of Padua, Padua, Italy

    Linda Capellato

  8. of Information Engineering (DEI), University of Padua, Department, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Goeuriot, L. et al. (2015). Overview of the CLEF eHealth Evaluation Lab 2015. In: Mothe, J., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2015. Lecture Notes in Computer Science(), vol 9283. Springer, Cham. https://doi.org/10.1007/978-3-319-24027-5_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24027-5_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24026-8

  • Online ISBN: 978-3-319-24027-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation