Skip to main content

Overview of the CLEF eHealth Evaluation Lab 2016

Part of the Lecture Notes in Computer Science book series (LNISA,volume 9822)

Abstract

In this paper we provide an overview of the fourth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2016 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins and clinical staff in understanding, accessing and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on handover information extraction related to Australian nursing shift changes, Task 2 on information extraction in French corpora, and Task 3 on multilingual patient-centred information retrieval considering query variations. In total 20 teams took part in these tasks (3 in Task 1, 7 in Task 2 and 10 in Task 3). Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants to this year’s challenges and some results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

Keywords

  • Evaluation
  • Entity linking
  • Information retrieval
  • Information extraction
  • Medical informatics
  • Nursing records
  • Patient handoff/handover
  • Speech recognition
  • Test-set generation
  • Text classification
  • Text segmentation
  • Self-diagnosis

In alphabetical order, LK & LG co-chaired the lab. HS led Task 1. AN led Task 2. JP & GZ led Task 3.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-44564-9_24
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-44564-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)

Notes

  1. 1.

    See our live demostration at http://nicta-stct.s3-website-ap-southeast-2.amazonaws.com, last accessed on 25 May 2016.

  2. 2.

    http://lemurproject.org/clueweb12/index.php.

  3. 3.

    The organisers are thankful to Carnegie Mellon University, and in particular to Jamie Callan and Christina Melucci, for their support in obtaining the permission to redistribute ClueWeb 12. The organisers are also thankful to Microsoft Azure who provided the Azure cloud computing infrastructure that was made available to participants through the Microsoft Azure for Research Award CRM:0518649.

  4. 4.

    https://www.reddit.com/r/AskDocs/.

  5. 5.

    See http://www.cnts.ua.ac.be/conll2000/chunking/, last accessed on 25 May 2016.

  6. 6.

    using https://www.random.org/, last accessed on 25 May 2016.

  7. 7.

    The persistence parameter p in RBP was set to 0.8.

  8. 8.

    https://github.com/ielab/ubire, [22].

  9. 9.

    https://sites.google.com/site/clefehealth/.

References

  1. Suominen, H., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 212–231. Springer, Heidelberg (2013)

    Google Scholar 

  2. Kelly, L., et al.: Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 172–191. Springer, Heidelberg (2014)

    Google Scholar 

  3. Goeuriot, L., Kelly, L., Suominen, H., Hanlen, L., Névéol, A., Grouin, C., Palotti, J., Zuccon, G.: Overview of the CLEF eHealth evaluation lab 2015. In: Mothe, J., Savoy, J., Kamps, J., Pinel-Sauvagnat, K., Jones, G.J.F., SanJuan, E., Cappellato, L., Ferro, N. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 429–443. Springer, Heidelberg (2015)

    Google Scholar 

  4. Suominen, H., Hanlen, L., Goeuriot, L., Kelly, L., Jones, G.J.: Task 1a of the CLEF eHealth evaluation lab 2015: Clinical speech recognition. In: CLEF 2015 Online Working Notes, CEUR-WS (2015)

    Google Scholar 

  5. Suominen, H., Johnson, M., Zhou, L., Sanchez, P., Sirel, R., Basilakis, J., Hanlen, L., Estival, D., Dawson, L., Kelly, B.: Capturing patient information at nursing shift changes: methodological evaluation of speech recognition and information extraction. J. Am. Med. Inform. Assoc. (JAMIA) 22(e1), e48–e66 (2015)

    Google Scholar 

  6. Névéol, A., Grouin, C., Leixa, J., Rosset, S., Zweigenbaum, P.: The QUAERO French medical corpus: a resource for medical entity recognition and normalization. In: Proceeding of BioTextMining Work, pp. 24–30 (2014)

    Google Scholar 

  7. Névéol, A., Grouin, C., Tannier, X., Hamon, T., Kelly, L., Goeuriot, L., Zweigenbaum, P.: CLEF eHealth evaluation lab 2015 task 1b: clinical named entity recognition. In: CLEF 2015 Online Working Notes, CEUR-WS (2015)

    Google Scholar 

  8. Goeuriot, L., Kelly, L., Li, W., Palotti, J., Pecina, P., Zuccon, G., Hanbury, A., Jones, G.J., Mueller, H.: ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: user-centred health information retrieval. In: CLEF 2014 Evaluation Labs and Workshop: Online Working Notes, Sheffield (2014)

    Google Scholar 

  9. Palotti, J., Zuccon, G., Goeuriot, L., Kelly, L., Hanburyn, A., Jones, G.J., Lupu, M., Pecina, P.: CLEF eHealth evaluation lab 2015, task 2: Retrieving information about medical symptoms. In: CLEF 2015 Online Working Notes, CEUR-WS (2015)

    Google Scholar 

  10. Suominen, H., Zhou, L., Hanlen, L., Ferraro, G.: Benchmarking clinical speech recognition and information extraction: new data, methods and evaluations. JMIR Med. Informatics 3, e19 (2015)

    CrossRef  Google Scholar 

  11. Suominen, H., Zhou, L., Goeuriot, L., Kelly, L.: Task 1 of the CLEF eHealth evaluation lab 2016: Handover information extraction. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS (2016)

    Google Scholar 

  12. Hanbury, A., Müller, H.: Khresmoi - multimodal multilingual medical information search. In: MIE village of the future (2012)

    Google Scholar 

  13. Goeuriot, L., Jones, G.J., Kelly, L., Leveling, J., Hanbury, A., Müller, H., Salantera, S., Suominen, H., Zuccon, G.: Share, CLEF eHealth evaluation lab 2013, task 3: Information retrieval to address patients’ questions when reading clinical reports. CLEF: Online Working Notes 8138 (2013)

    Google Scholar 

  14. Macdonald, C., McCreadie, R., Santos, R.L., Ounis, I.: From puppy to maturity: experiences in developing terrier. In: Proceding of OSIR at SIGIR, pp. 60–63 (2012)

    Google Scholar 

  15. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis. vol. 2, pp. 2–6, Citeseer (2005)

    Google Scholar 

  16. Cormack, G.V., Smucker, M.D., Clarke, C.L.: Efficient and effective spam filtering and re-ranking for large web datasets. Inform. Retrieval 14, 441–465 (2011)

    CrossRef  Google Scholar 

  17. Hiemstra, D., Hauff, C.: Mirex: Mapreduce information retrieval experiments. arXiv preprint arXiv:1004.4489 (2010)

  18. Bodenreider, O., McCray, A.T.: Exploring semantic groups through visual approaches. J. Biomed. Inform. 36, 414–432 (2003)

    CrossRef  Google Scholar 

  19. Bailey, P., Moffat, A., Scholer, F., Thomas, P.: User Variability and IR System Evaluation. In: Proceeding of SIGIR (2015)

    Google Scholar 

  20. Azzopardi, L.: Query side evaluation: an empirical analysis of effectiveness and effort. In: Proceeding of SIGIR (2009)

    Google Scholar 

  21. Zuccon, G., Koopman, B.: Integrating understandability in the evaluation of consumer health search engines. In: Medical Information Retrieval Workshop at SIGIR 2014, p. 32 (2014)

    Google Scholar 

  22. Zuccon, G.: Understandability biased evaluation for information retrieval. In: Advances in Information Retrieval, pp. 280–292 (2016)

    Google Scholar 

  23. Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1, 80–83 (1945)

    CrossRef  Google Scholar 

  24. Zuccon, G., Palotti, J., Goeuriot, L., Kelly, L., Lupu, M., Pecina, P., Mueller, H., Budaher, J., Deacon, A.: The IR Task at the CLEF eHealth evaluation lab 2016: user-centred health information retrieval. In: CLEF 2016 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS (2016)

    Google Scholar 

  25. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inform. Syst. 20, 422–446 (2002)

    CrossRef  Google Scholar 

Download references

Acknowledgements

The CLEF eHealth 2016 evaluation lab has been supported in part by (in alphabetical order) the ANR, the French National Research Agency, under grant CABeRneT ANR-13-JS02-0009-01; CLEF Initiative; ESF ELIAS network program Horizon 2020 program (H2020-ICT-2014-1) under grant agreement 644753 (KCONNECT); Microsoft Azure for Research Award CRM:0518649; NICTA, funded by the Australian Government through the Department of Communications and the Australian Research Council through the Information and Communications Technology (ICT) Centre of Excellence Program; and PhysioNetWorks Workspaces.

We express our gratitude to Maricel Angel, Registered Nurse at NICTA, for helping us to create the Task 1 dataset, using the Protégé resource, which is supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States National Institutes of Health. We are also thankful to the people involved in the query creation and relevance assessment exercise. Last but not least, we gratefully acknowledge the participating teams’ hard work. We thank them for their submissions and interest in the lab.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Liadh Kelly .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Kelly, L., Goeuriot, L., Suominen, H., Névéol, A., Palotti, J., Zuccon, G. (2016). Overview of the CLEF eHealth Evaluation Lab 2016. In: , et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2016. Lecture Notes in Computer Science(), vol 9822. Springer, Cham. https://doi.org/10.1007/978-3-319-44564-9_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-44564-9_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-44563-2

  • Online ISBN: 978-3-319-44564-9

  • eBook Packages: Computer ScienceComputer Science (R0)