Skip to main content
SpringerLink
Log in
Book cover

International Conference of the Cross-Language Evaluation Forum for European Languages

CLEF 2017: Experimental IR Meets Multilinguality, Multimodality, and Interaction pp 291–303Cite as

CLEF 2017 eHealth Evaluation Lab Overview

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10456))

Abstract

In this paper we provide an overview of the fifth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2017 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on multilingual information extraction to extend from last year’s task on French corpora, Task 2 on technologically assisted reviews in empirical medicine as a new pilot task, and Task 3 on patient-centered information retrieval (IR) building on the 2013-16 IR tasks. In total 32 teams took part in these tasks (11 in Task 1, 14 in Task 2, and 7 in Task 3). We also continued the replication track from 2016. Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants of this year’s challenges and results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

G. Zuccon—In alphabetical order by surname, LG, LK & HS co-chaired the lab. AN & AR, EK & RS, and JP & GZ led Tasks 1–3, respectively.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.cochrane.org/.

  2. 2.

    http://www.cochranelibrary.com.

  3. 3.

    http://ovid.com/site/catalog/databases/901.jsp.

  4. 4.

    https://www.ncbi.nlm.nih.gov/pubmed/.

  5. 5.

    http://lemurproject.org/clueweb12/index.php.

  6. 6.

    The organisers are thankful to Carnegie Mellon University, and in particular to Jamie Callan and Christina Melucci, for their support in obtaining the permission to redistribute ClueWeb 12. The organisers are also thankful to Microsoft Azure who provided the Azure cloud computing infrastructure that was made available to participants through the Microsoft Azure for Research Award CRM:0518649.

  7. 7.

    https://www.reddit.com/r/AskDocs/.

  8. 8.

    The persistence parameter p in RBP was set to 0.8.

  9. 9.

    https://github.com/ielab/ubire, [20].

  10. 10.

    See Task 1 paper for details on unofficial runs [14].

  11. 11.

    https://sites.google.com/site/clefehealth/.

References

  1. Cormack, G.V., Grossman, M.R.: Engineering quality and reliability in technology-assisted review. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016, pp. 75–84. ACM, New York (2016). http://doi.acm.org/10.1145/2911451.2911510

  2. Cormack, G.V., Smucker, M.D., Clarke, C.L.: Efficient and effective spam filtering and re-ranking for large web datasets. Inf. Retrieval 14(5), 441–465 (2011)

    Article  Google Scholar 

  3. Goeuriot, L., Jones, G.J., Kelly, L., Leveling, J., Hanbury, A., Müller, H., Salantera, S., Suominen, H., Zuccon, G.: ShARe, CLEF eHealth Evaluation Lab 2013, Task 3: Information retrieval to address patients’ questions when reading clinical reports. CLEF 2013 Online Working. Notes 8138 (2013)

    Google Scholar 

  4. Goeuriot, L., Kelly, L., Lee, W., Palotti, J., Pecina, P., Zuccon, G., Hanbury, A., Gareth J.F. Jones, H.M.: ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: user-centred health information retrieval. In: CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield, UK (2014)

    Google Scholar 

  5. Goeuriot, L., Kelly, L., Suominen, H., Hanlen, L., Névéol, A., Grouin, C., Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2015. In: Mothe, J., Savoy, J., Kamps, J., Pinel-Sauvagnat, K., Jones, G.J.F., SanJuan, E., Cappellato, L., Ferro, N. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 429–443. Springer, Cham (2015). doi:10.1007/978-3-319-24027-5_44

    Chapter  Google Scholar 

  6. Hanbury, A., Müller, H.: Khresmoi - multimodal multilingual medical information search. In: Medical Informatics Europe 2012 (MIE 2012), Village of the Future (2012)

    Google Scholar 

  7. Hiemstra, D., Hauff, C.: Mirex: Mapreduce information retrieval experiments. arXiv preprint arXiv:1004.4489

  8. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)

    Article  Google Scholar 

  9. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: Overview of the CLEF technologically assisted reviews in empirical medicine. In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings (2017)

    Google Scholar 

  10. Kelly, L., Goeuriot, L., Suominen, H., Névéol, A., Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2016. In: Fuhr, N., Quaresma, P., Gonçalves, T., Larsen, B., Balog, K., Macdonald, C., Cappellato, L., Ferro, N. (eds.) CLEF 2016. LNCS, vol. 9822, pp. 255–266. Springer, Cham (2016). doi:10.1007/978-3-319-44564-9_24

    Chapter  Google Scholar 

  11. Kelly, L., Goeuriot, L., Suominen, H., Schreck, T., Leroy, G., Mowery, D.L., Velupillai, S., Chapman, W.W., Martinez, D., Zuccon, G., Palotti, J.: Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 172–191. Springer, Cham (2014). doi:10.1007/978-3-319-11382-1_17

    Google Scholar 

  12. Leeflang, M.M., Deeks, J.J., Takwoingi, Y., Macaskill, P.: Cochrane diagnostic test accuracy reviews. Syst. Rev. 2(1), 82 (2013)

    Article  Google Scholar 

  13. Macdonald, C., McCreadie, R., Santos, R.L., Ounis, I.: From puppy to maturity: experiences in developing terrier. In: Proceeding of OSIR at SIGIR, pp. 60–63 (2012)

    Google Scholar 

  14. Névéol, A., Anderson, R.N., Cohen, K.B., Grouin, C., Lavergne, T., Rey, G., Robert, A., Zweigenbaum, P.: CLEF eHealth 2017 multilingual information extraction task overview: ICD10 coding of death certificates in English and French. In: CLEF 2017 Online Working Notes. CEUR-WS (2017)

    Google Scholar 

  15. Névéol, A., Cohen, K.B., Grouin, C., Hamon, T., Lavergne, T., Kelly, L., Goeuriot, L., Rey, G., Robert, A., Tannier, X., Zweigenbaum, P.: Clinical information extraction at the CLEF eHealth evaluation lab 2016. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS (2016)

    Google Scholar 

  16. Palotti, J., Zuccon, G., Goeuriot, L., Kelly, L., Hanburyn, A., Jones, G.J., Lupu, M., Pecina, P.: CLEF eHealth evaluation lab 2015, Task 2: retrieving information about medical symptoms. In: CLEF 2015 Online Working Notes. CEUR-WS (2015)

    Google Scholar 

  17. Palotti, J., Zuccon, G., Jimmy, Pecina, P., Lupu, M., Goeuriot, L., Kelly, L., Hanbury, A.: CLEF 2017 task overview: the IR task at the eHealth evaluation lab. In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings (2017)

    Google Scholar 

  18. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis, vol. 2, pp. 2–6. Citeseer (2005)

    Google Scholar 

  19. Suominen, H., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 212–231. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40802-1_24

    Chapter  Google Scholar 

  20. Zuccon, G.: Understandability biased evaluation for information retrieval. In: Advances in Information Retrieval, pp. 280–292 (2016)

    Google Scholar 

  21. Zuccon, G., Koopman, B.: Integrating understandability in the evaluation of consumer health search engines. In: Medical Information Retrieval Workshop at SIGIR 2014, p. 32 (2014)

    Google Scholar 

  22. Zuccon, G., Palotti, J., Goeuriot, L., Kelly, L., Lupu, M., Pecina, P., Mueller, H., Budaher, J., Deacon, A.: The IR task at the CLEF eHealth evaluation lab 2016: user-centred health information retrieval. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS, September 2016

    Google Scholar 

Download references

Acknowledgements

The CLEF eHealth 2017 evaluation lab has been supported in part by (in alphabetical order) the ANR, the French National Research Agency, under grant CABeRneT ANR-13-JS02-0009-01, the CLEF Initiative and Data61. We are also thankful to the people involved in the annotation, query creation, and relevance assessment exercise. Last but not least, we gratefully acknowledge the participating teams’ hard work. We thank them for their submissions and interest in the lab.

Author information

Authors and Affiliations

  1. University Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000, Grenoble, France

    Lorraine Goeuriot

  2. ADAPT Centre, Dublin City University, Dublin, Ireland

    Liadh Kelly

  3. Data61/CSIRO, The Australian National University (ANU), Canberra, ACT, Australia

    Hanna Suominen

  4. University of Canberra, Canberra, ACT, Australia

    Hanna Suominen

  5. University of Turku, Turku, Finland

    Hanna Suominen

  6. LIMSI CNRS UPR 3251 Université Paris-Saclay, 91405, Orsay, France

    Aurélie Névéol

  7. INSERM - CépiDc, 80 Rue du Général Leclerc, 94276, Le Kremlin-Bicêtre Cedex, France

    Aude Robert

  8. Informatics Institute, University of Amsterdam, Amsterdam, Netherlands

    Evangelos Kanoulas

  9. Julius Center for Health Sciences and Primary Care, Cochrane Netherlands and UMC Utrecht, Utrecht, Netherlands

    Rene Spijker

  10. Vienna University of Technology, Vienna, Austria

    João Palotti

  11. Queensland University of Technology, Brisbane, QLD, Australia

    Guido Zuccon

Authors
  1. Lorraine Goeuriot
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liadh Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanna Suominen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aurélie Névéol
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aude Robert
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Evangelos Kanoulas
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rene Spijker
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. João Palotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Guido Zuccon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanna Suominen .

Editor information

Editors and Affiliations

  1. Dublin City University, Dublin, Ireland

    Gareth J.F. Jones

  2. Trinity College Dublin, Dublin, Ireland

    Séamus Lawless

  3. National University of Distance Education, Madrid, Spain

    Julio Gonzalo

  4. Dublin City University, Dublin, Ireland

    Liadh Kelly

  5. Université Grenoble Alpes, Grenoble, France

    Lorraine Goeuriot

  6. University of Hildesheim, Hildesheim, Germany

    Thomas Mandl

  7. University of Padua, Padua, Italy

    Linda Cappellato

  8. University of Padua, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Goeuriot, L. et al. (2017). CLEF 2017 eHealth Evaluation Lab Overview. In: Jones, G., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2017. Lecture Notes in Computer Science(), vol 10456. Springer, Cham. https://doi.org/10.1007/978-3-319-65813-1_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-65813-1_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-65812-4

  • Online ISBN: 978-3-319-65813-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation