Skip to main content
SpringerLink
Log in

CLEF eHealth Evaluation Lab 2021

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12657))

Included in the following conference series:

Abstract

Motivated by the ever increasing difficulties faced by laypeople in retrieving and digesting valid and relevant information to make health-centred decisions, the CLEF eHealth lab series has offered shared tasks to the community in the fields of Information Extraction (IE), management, and Information Retrieval (IR) since 2013. These tasks have attracted large participation and led to statistically significant improvements in processing quality. In 2021, CLEF eHealth is calling for participants to contribute to the following two tasks: Task 1 on IE focuses on IE from noisy text. Participants will identify and classify Named Entities in written ultrasonography reports, containing misspellings and inconsistencies, from a major public hospital in Argentina. Identified entities will then have to be classified, which can be very challenging as it requires to handle lexical variations. Task 2 is a novel extension of the most popular and established task on consumer health search (CHS), aiming at retrieving relevant, understandable, and credible information for patients and their next-of-kins. In this paper we describe recent advances in the fields of IE and IR, and the subsequent offerings of this years CLEF eHealth lab challenges.

LG, HS, & LK co-chair the CLEF eHealth lab and contributed equally to this paper. Task 1 is led by LAA and VC, and organized by DF, FL, RR, and JV; Task 2 is led by LG, GP, and HS, and organized by NB-S, GGS, LK, PM, SS, MV, and CX.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In the paper, we consider patients, layperson or consumer, to be system users with no or little medical background.

  2. 2.

    http://clefehealth.imag.fr.

  3. 3.

    https://commoncrawl.org/.

References

  1. Bossy, R., Golik, W., Ratkovic, Z., Bessières, P., Nédellec, C.: BioNLP shared task 2013 - an overview of the bacteria biotope task. In: Proceedings of the BioNLP Shared Task 2013 Workshop. pp. 161–169. Association for Computational Linguistics, Sofia, Bulgaria, August 2013. https://www.aclweb.org/anthology/W13-2024

  2. Bruns, A.: Echo chamber? What echo chamber? Reviewing the evidence (2017)

    Google Scholar 

  3. Ceravolo, P., Damiani, E., Viviani, M.: Adding a trust layer to semantic web metadata. In: Herrera-Viedma, E., Pasi, G., Crestani, F. (eds.) Soft Computing in Web Information Retrieval. Studies in Fuzziness and Soft Computing, vol. 197, pp. 87–104. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-31590-X_5

    Chapter  Google Scholar 

  4. Demner-Fushman, D., Elhadad, N.: Aspiring to unintended consequences of natural language processing: a review of recent developments in clinical and consumer-generated text processing. Yearb. Med. Inform. 1, 224–233 (2016)

    Google Scholar 

  5. Filannino, M., Uzuner, Ö.: Advancing the state of the art in clinical natural language processing through shared tasks. Yearb. Med. Inform. 27(01), 184–192 (2018)

    Article  Google Scholar 

  6. Goeuriot, L., et al.: An analysis of evaluation campaigns in ad-hoc medical information retrieval: CLEF eHealth 2013 and 2014. Inf. Retr. J. 21, 507–540 (2018). https://doi.org/10.1007/s10791-018-9331-4

    Article  Google Scholar 

  7. Goeuriot, L., et al.: Overview of the CLEF eHealth evaluation lab 2015. In: Mothe, J., et al. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 429–443. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24027-5_44

    Chapter  Google Scholar 

  8. Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 291–303. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_26

    Chapter  Google Scholar 

  9. Goeuriot, L., Liu, Z., Pasi, G., Saez, G.G., Viviani, M., Xu, C.: Overview of the CLEF eHealth 2020 task 2: consumer health search with ad hoc and spoken queries. In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings (2020)

    Google Scholar 

  10. Goeuriot, L., et al.: Overview of the CLEF eHealth evaluation lab 2020. In: Arampatzis, A., et al. (eds.) CLEF 2020. LNCS, vol. 12260, pp. 255–271. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58219-7_19

    Chapter  Google Scholar 

  11. Huang, C.C., Lu, Z.: Community challenges in biomedical text mining over 10 years: success, failure and the future. Brief. Bioinform. 17(1), 132–144 (2016)

    Article  Google Scholar 

  12. Kelly, L., et al.: Overview of the CLEF eHealth evaluation lab 2016. In: Fuhr, N., et al. (eds.) CLEF 2016. LNCS, vol. 9822, pp. 255–266. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44564-9_24

    Chapter  Google Scholar 

  13. Kelly, L., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Kanoulas, E., et al. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 172–191. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11382-1_17

    Chapter  Google Scholar 

  14. Kelly, L., et al.: Overview of the CLEF eHealth evaluation lab 2019. In: Crestani, F., et al. (eds.) CLEF 2019. LNCS, vol. 11696, pp. 322–339. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28577-7_26

    Chapter  Google Scholar 

  15. Makhoul, J., Kubala, F., Schwartz, R., Weischedel, R.: Performance measures for information extraction. In: Proceedings of DARPA Broadcast News Workshop, pp. 249–252 (1999)

    Google Scholar 

  16. McAllister, M., Dunn, G., Payne, K., Davies, L., Todd, C.: Patient empowerment: the need to consider it as a measurable patient-reported outcome for chronic conditions. BMC Health Serv. Res. 12, 157 (2012)

    Article  Google Scholar 

  17. Mukherjee, S., Weikum, G., Danescu-Niculescu-Mizil, C.: People on drugs: credibility of user statements in health communities. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 65–74 (2014)

    Google Scholar 

  18. Self, C.C.: Credibility. In: An Integrated Approach to Communication Theory and Research, pp. 449–470. Routledge (2014)

    Google Scholar 

  19. Suominen, H.: CLEFeHealth2012 – the CLEF 2012 workshop on cross-language evaluation of methods, applications, and resources for eHealth document analysis. In: Forner, P., Karlgren, J., Womser-Hacker, C., Ferro, N. (eds.) CLEF 2012 Working Notes. vol. 1178. CEUR Workshop Proceedings (CEUR-WS.org) (2012)

    Google Scholar 

  20. Suominen, H., Kelly, L., Goeuriot, L.: Scholarly influence of the conference and labs of the evaluation forum eHealth initiative: review and bibliometric study of the 2012 to 2017 outcomes. JMIR Res. Prot. 7(7), e10961 (2018)

    Google Scholar 

  21. Suominen, H., Kelly, L., Goeuriot, L.: The scholarly impact and strategic intent of CLEF eHealth labs from 2012 to 2017. Information Retrieval Evaluation in a Changing World. TIRS, vol. 41, pp. 333–363. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22948-1_14

    Chapter  Google Scholar 

  22. Suominen, H., et al.: Overview of the CLEF eHealth evaluation lab 2018. In: Bellot, P., et al. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. LNCS, vol. 11018, pp. 286–301. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_26

    Chapter  Google Scholar 

  23. Suominen, H., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 212–231. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40802-1_24

    Chapter  Google Scholar 

  24. Viviani, M., Pasi, G.: A multi-criteria decision making approach for the assessment of information credibility in social media. In: Petrosino, A., Loia, V., Pedrycz, W. (eds.) WILF 2016. LNCS (LNAI), vol. 10147, pp. 197–207. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52962-2_17

    Chapter  Google Scholar 

  25. Viviani, M., Pasi, G.: Credibility in social media: opinions, news, and health information–a survey. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 7(5), e1209 (2017)

    Google Scholar 

Download references

Acknowledgements

The lab has been supported in part by the CLEF Initiative and the Our Health in Our Hands (OHIOH) initiative of The Australian National University (ANU). OHIOH is a strategic initiative of The ANU which aims to transform healthcare by developing new personalised health technologies and solutions in collaboration with patients, clinicians, and healthcare providers. The lab has been supported in part by the bi-lateral Kodicare (Knowledge Delta based improvement and continuous evaluation of retrieval engines) project funded by the french ANR (ANR-19-CE23-0029) and Austrian FWF.

Author information

Authors and Affiliations

  1. Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000, Grenoble, France

    Lorraine Goeuriot, Gabriela Gonzalez Saez & Philippe Mulhem

  2. The Australian National University (ANU), Data61/Commonwealth Scientific and Industrial Research Organisation (CSIRO), and University of Turku, Canberra, ACT, Australia

    Hanna Suominen

  3. Maynooth University, Co., Kildare, Ireland

    Liadh Kelly

  4. Universidad Nacional de Córdoba, Córdoba, Argentina

    Laura Alonso Alemany

  5. The ANU, Canberra, ACT, Australia

    Nicola Brew-Sam & Sandaru Seneviratne

  6. Universidad de Buenos Aires, CONICET, Buenos Aires, Argentina

    Viviana Cotik

  7. Hospital de Pediatría ‘Prof. Dr. Juan P. Garrahan’, Buenos Aires, Argentina

    Darío Filippo

  8. University of Milano-Bicocca, Milan, Italy

    Gabriella Pasi & Marco Viviani

  9. German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

    Roland Roller

  10. Institut de Lingüística Aplicada, Universitat Pompeu Fabra, Barcelona, Spain

    Jorge Vivaldi

  11. The ANU and Data61/CSIRO, Canberra, ACT, Australia

    Chenchen Xu

  12. Universidad Nacional de Córdoba, CONICET, Córdoba, Argentina

    Franco Luque

Authors
  1. Lorraine Goeuriot
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanna Suominen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liadh Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laura Alonso Alemany
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nicola Brew-Sam
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Viviana Cotik
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Darío Filippo
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gabriela Gonzalez Saez
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Franco Luque
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Philippe Mulhem
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Gabriella Pasi
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Roland Roller
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Sandaru Seneviratne
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Jorge Vivaldi
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Marco Viviani
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Chenchen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lorraine Goeuriot .

Editor information

Editors and Affiliations

  1. Radboud University Nijmegen, Nijmegen, The Netherlands

    Djoerd Hiemstra

  2. Department of Computer Science, Katholieke Universiteit Leuven, Heverlee, Belgium

    Marie-Francine Moens

  3. Toulouse, Toulouse Institute of Computer Science Research, Toulouse, France

    Josiane Mothe

  4. Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Raffaele Perego

  5. Leipzig University, Leipzig, Germany

    Martin Potthast

  6. Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Fabrizio Sebastiani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Goeuriot, L. et al. (2021). CLEF eHealth Evaluation Lab 2021. In: Hiemstra, D., Moens, MF., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2021. Lecture Notes in Computer Science(), vol 12657. Springer, Cham. https://doi.org/10.1007/978-3-030-72240-1_69

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72240-1_69

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72239-5

  • Online ISBN: 978-3-030-72240-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation