Skip to main content
SpringerLink
Log in

Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11018))

Abstract

This paper presents an overview of the ImageCLEF 2018 evaluation campaign, an event that was organized as part of the CLEF (Conference and Labs of the Evaluation Forum) Labs 2018. ImageCLEF is an ongoing initiative (it started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval with the aim of providing information access to collections of images in various usage scenarios and domains. In 2018, the 16th edition of ImageCLEF ran three main tasks and a pilot task: (1) a caption prediction task that aims at predicting the caption of a figure from the biomedical literature based only on the figure image; (2) a tuberculosis task that aims at detecting the tuberculosis type, severity and drug resistance from CT (Computed Tomography) volumes of the lung; (3) a LifeLog task (videos, images and other sources) about daily activities understanding and moment retrieval, and (4) a pilot task on visual question answering where systems are tasked with answering medical questions. The strong participation, with over 100 research groups registering and 31 submitting results for the tasks, shows an increasing interest in this benchmarking campaign.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://imageclef.org/2018/.

  2. 2.

    http://clef2018.clef-initiative.eu/.

  3. 3.

    https://www.crowdai.org/.

  4. 4.

    https://www.ncbi.nlm.nih.gov/pmc/.

  5. 5.

    https://keras.io/.

  6. 6.

    https://en.wikipedia.org/wiki/Cohen’s_kappa.

  7. 7.

    http://www.visualqa.org/.

  8. 8.

    https://www.ncbi.nlm.nih.gov/pmc/.

  9. 9.

    http://www.cs.cmu.edu/~ark/mheilman/questions/.

  10. 10.

    There was a limit of maximum 5 run submissions per team.

  11. 11.

    http://www.imageclef.org/2017/caption.

  12. 12.

    https://datasets.d2.mpi-inf.mpg.de/mateusz14visualturing/calculate_wups.py.

  13. 13.

    https://metamap.nlm.nih.gov/.

  14. 14.

    https://github.com/AnthonyMRios/pymetamap.

  15. 15.

    http://www.jcdl.org/archived-conf-sites/jcdl2015/www.jcdl2015.org/panels.html.

  16. 16.

    http://irlld2016.computing.dcu.ie/index.html.

  17. 17.

    http://lta2016.computing.dcu.ie.

  18. 18.

    http://lta2017.computing.dcu.ie.

  19. 19.

    http://ntcir-lifelog.computing.dcu.ie.

References

  1. Abdallah, F.B., Feki, G., Ezzarka, M., Ammar, A.B., Amar, C.B.: Regim Lab Team at ImageCLEFlifelog LMRT Task 2018, 10–14 September 2018

    Google Scholar 

  2. Andrearczyk, V., Henning, M.: Deep multimodal classification of image types in biomedical journal figures. In: Ferro, N., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 3–14. Springer, Cham (2018)

    Google Scholar 

  3. Antol, S., et al.: VQA: visual question answering. In: International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

  4. Balikas, G., Krithara, A., Partalas, I., Paliouras, G.: BioASQ: a challenge on large-scale biomedical semantic indexing and question answering. In: Müller, H., Jimenez del Toro, O.A., Hanbury, A., Langs, G., Foncubierta Rodríguez, A. (eds.) MRDM 2015. LNCS, vol. 9059, pp. 26–39. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24471-6_3

    Chapter  Google Scholar 

  5. Clough, P., Müller, H., Sanderson, M.: The CLEF 2004 cross-language image retrieval track. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 597–613. Springer, Heidelberg (2005). https://doi.org/10.1007/11519645_59

    Chapter  Google Scholar 

  6. Dang-Nguyen, D.T., Piras, L., Riegler, M., Boato, G., Zhou, L., Gurrin, C.: Overview of ImageCLEFlifelog 2017: lifelog retrieval and summarization. In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017). http://ceur-ws.org

  7. Dang-Nguyen, D.T., Piras, L., Riegler, M., Zhou, L., Lux, M., Gurrin, C.: Overview of ImageCLEFlifelog 2018: daily living understanding and lifelog moment retrieval. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  8. Dao, M.S., Kasem, A., Nazmudeen, M.S.H.: Leveraging Content and Context to Foster Understanding of Activities of Daily Living, 10–14 September 2018

    Google Scholar 

  9. Dicente Cid, Y., Jimenez-del-Toro, O., Depeursinge, A., Müller, H.: Efficient and fully automatic segmentation of the lungs in CT volumes. In: Goksel, O., Jimenez-del-Toro, O., Foncubierta-Rodriguez, A., Müller, H. (eds.) Proceedings of the VISCERAL Challenge at ISBI. No. 1390 in CEUR Workshop Proceedings, April 2015

    Google Scholar 

  10. Dicente Cid, Y., Kalinovsky, A., Liauchuk, V., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2017 - predicting tuberculosis type and drug resistances. In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017). http://ceur-ws.org

  11. Dicente Cid, Y., Liauchuk, V., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2018 - detecting multi-drug resistance, classifying tuberculosis type, and assessing severity score. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  12. Dogariu, M., Ionescu, B.: Multimedia Lab @ CAMPUS at ImageCLEFlifelog 2018 Lifelog Moment Retrieval, 10–14 September 2018

    Google Scholar 

  13. Eickhoff, C., Schwall, I., García Seco de Herrera, A., Müller, H.: Overview of ImageCLEFcaption 2017 - image caption prediction and concept detection for biomedical images. In: CLEF 2017 Labs Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017). http://ceur-ws.org

  14. Goeuriot, L., et al.: CLEF 2017 eHealth evaluation lab overview. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 291–303. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_26

    Chapter  Google Scholar 

  15. Gravier, G., et al.: Working notes proceedings of the mediaeval 2017 workshop. In: MediaEval 2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, 13–15 September 2017. CEUR-WS.org (2017). http://ceur-ws.org

  16. Gurrin, C., et al.: Overview of NTCIR-13 Lifelog-2 task. In: Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies (2017)

    Google Scholar 

  17. Hanbury, A., et al.: Evaluation-as-a-service: overview and outlook. ArXiv arXiv:1512.07454 (2015)

  18. Hanbury, A., Müller, H., Langs, G., Weber, M.A., Menze, B.H., Fernandez, T.S.: Bringing the algorithms to the data: cloud–based benchmarking for medical image analysis. In: Catarci, T., Forner, P., Hiemstra, D., Peñas, A., Santucci, G. (eds.) CLEF 2012. LNCS, vol. 7488, pp. 24–29. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33247-0_3

    Chapter  Google Scholar 

  19. Hasan, S.A., Ling, Y., Farri, O., Liu, J., Lungren, M., Müller, H.: Overview of the ImageCLEF 2018 medical domain visual question answering task. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  20. García Seco de Herrera, A., Eickhoff, C., Andrearczyk, V., Müller, H.: Overview of the ImageCLEF 2018 caption prediction tasks. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  21. Ionescu, B., et al.: Overview of ImageCLEF 2017: information extraction from images. In: Jones, G.J.F., et al. (eds.) CLEF 2017. LNCS, vol. 10456, pp. 315–337. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1_28

    Chapter  Google Scholar 

  22. Joly, A., et al.: LifeCLEF 2017 lab overview: multimedia species identification challenges. In: Proceedings of CLEF 2017 (2017)

    Google Scholar 

  23. Kalpathy-Cramer, J., García Seco de Herrera, A., Demner-Fushman, D., Antani, S., Bedrick, S., Müller, H.: Evaluating performance of biomedical image retrieval systems: overview of the medical image retrieval task at ImageCLEF 2004–2014. Comput. Med. Imaging Graph. 39, 55–61 (2015)

    Article  Google Scholar 

  24. Kavallieratou, E., del Blanco, C.R., Cuevas, C., García, N.: Retrieving Events in Life Logging, 10–14 September 2018

    Google Scholar 

  25. Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF - Experimental Evaluation in Visual Information Retrieval. Information Retrieval Series, vol. 32. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15181-1

    Book  MATH  Google Scholar 

  26. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics (2002)

    Google Scholar 

  27. Pinho, E., Costa, C.: Feature learning with adversarial networks for concept detection in medical images: UA.PT Bioinformatics at ImageCLEF 2018. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  28. Rahman, M.M.: A cross modal deep learning based approach for caption prediction and concept detection by CS Morgan State. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  29. Sharma, A., et al.: Estimating the future burden of multidrug-resistant and extensively drug-resistant tuberculosis in India, the Philippines, Russia, and South Africa: a mathematical modelling study. Lancet Infect. Dis. 17(7), 707–715 (2017). http://www.sciencedirect.com/science/article/pii/S1473309917302475

    Article  Google Scholar 

  30. Soldaini, L., Goharian, N.: QuickUMLS: a fast, unsupervised approach for medical concept extraction. In: MedIR Workshop, SIGIR (2016)

    Google Scholar 

  31. Soğancıoğlu, G., Öztürk, H., Özgür, A.: BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics 33(14), i49–i58 (2017)

    Article  Google Scholar 

  32. Spinks, G., Moens, M.F.: Generating text from images in a smooth representation space. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  33. Su, Y., Liu, F.: UMass at ImageCLEF caption prediction 2018 task. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  34. Tang, T.H., Fu, M.H., Huang, H.H., Chen, K.T., Chen, H.H.: NTU NLP-Lab at ImageCLEFlifelog 2018: Visual Concept Selection with Textual Knowledge for Understanding Activities of Daily Living and Life Moment Retrieval, 10–14 September 2018

    Google Scholar 

  35. Tran, M.T., Truong, T.D., Dinh-Duy, T., Vo-Ho, V.K., Luong, Q.A., Nguyen, V.T.: Lifelog Moment Retrieval with Visual Concept Fusion and Text-based Query Expansion, 10–14 September 2018

    Google Scholar 

  36. Tsikrika, T., de Herrera, A.G.S., Müller, H.: Assessing the scholarly impact of ImageCLEF. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 95–106. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23708-9_12

    Chapter  Google Scholar 

  37. Tsikrika, T., Larsen, B., Müller, H., Endrullis, S., Rahm, E.: The scholarly impact of CLEF (2000–2009). In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 1–12. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40802-1_1

    Chapter  Google Scholar 

  38. Valavanis, L., Kalamboukis, T.: IPL at ImageCLEF 2018: a kNN-based concept detection approach. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  39. Villegas, M., et al.: General overview of ImageCLEF at the CLEF 2015 labs. In: Mothe, J., et al. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 444–461. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24027-5_45

    Chapter  Google Scholar 

  40. Villegas, M., et al.: General overview of ImageCLEF at the CLEF 2016 labs. In: Fuhr, N., et al. (eds.) CLEF 2016. LNCS, vol. 9822, pp. 267–285. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44564-9_25

    Chapter  Google Scholar 

  41. Wang, X., Zhang, Y., Guo, Z., Li, J.: ImageSem at ImageCLEF 2018 caption task: image retrieval and transfer learning. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, Avignon, France, 10–14 September 2018. CEUR-WS.org (2018). http://ceur-ws.org

  42. Wang, Y.X.J., Chung, M.J., Skrahin, A., Rosenthal, A., Gabrielian, A., Tartakovsky, M.: Radiological signs associated with pulmonary multi-drug resistant tuberculosis: an analysis of published evidences. Quant. Imaging Med. Surg. 8(2), 161–173 (2018)

    Article  Google Scholar 

  43. Wu, Z., Palmer, M.: Verbs semantics and lexical selection. In: Proceedings of the 32nd Annual Meeting on Association for Computational Linguistics, pp. 133–138. Association for Computational Linguistics (1994)

    Google Scholar 

  44. Zhao, J.J., Kim, Y., Zhang, K., Rush, A.M., LeCun, Y.: Adversarially regularized autoencoders for generating discrete structures. CoRR, abs/1706.04223 (2017)

    Google Scholar 

  45. Zhou, L., Piras, L., Riegler, M., Lux, M., Dang-Nguyen1, D.T., Gurrin, C.: An interactive lifelog retrieval system for activities of daily living understanding, 10–14 September 2018

    Google Scholar 

Download references

Acknowledgements

Bogdan Ionescu—part of this work was supported by the Ministry of Innovation and Research, UEFISCDI, project SPIA-VA, agreement 2SOL/2017, grant PN-III-P2-2.1-SOL-2016-02-0002.

Duc-Tien Dang-Nguyen, Liting Zhou and Cathal Gurrin—part of this work has emanated from research supported in part by research grants from the Irish Research Council (IRC) under Grant Number GOIPG/2016/741 and Science Foundation Ireland (SFI) under grant number SFI/12/RC/2289.

Author information

Authors and Affiliations

  1. University Politehnica of Bucharest, Bucharest, Romania

    Bogdan Ionescu

  2. University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland

    Henning Müller, Vincent Andrearczyk & Yashin Dicente Cid

  3. omni:us, Berlin, Germany

    Mauricio Villegas

  4. University of Essex, Colchester, UK

    Alba García Seco de Herrera

  5. Brown University, Providence, RI, USA

    Carsten Eickhoff

  6. United Institute of Informatics Problems, Minsk, Belarus

    Vitali Liauchuk & Vassili Kovalev

  7. Artificial Intelligence Lab, Philips Research North America, Cambridge, MA, USA

    Sadid A. Hasan, Yuan Ling, Oladimeji Farri & Joey Liu

  8. Department of Radiology, Stanford University, Stanford, CA, USA

    Matthew Lungren

  9. Dublin City University, Dublin, Ireland

    Duc-Tien Dang-Nguyen, Liting Zhou & Cathal Gurrin

  10. University of Cagliari and Pluribus One, Cagliari, Italy

    Luca Piras

  11. University of Oslo, Oslo, Norway

    Michael Riegler

  12. Simula Metropolitan Center for Digital Engineering, Oslo, Norway

    Michael Riegler

  13. Klagenfurt University, Klagenfurt, Austria

    Mathias Lux

Authors
  1. Bogdan Ionescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henning Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mauricio Villegas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alba García Seco de Herrera
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carsten Eickhoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vincent Andrearczyk
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yashin Dicente Cid
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Vitali Liauchuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Vassili Kovalev
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sadid A. Hasan
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Yuan Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Oladimeji Farri
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Joey Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Matthew Lungren
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Duc-Tien Dang-Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Luca Piras
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Michael Riegler
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Liting Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Mathias Lux
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Cathal Gurrin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bogdan Ionescu .

Editor information

Editors and Affiliations

  1. Aix-Marseille University, Marseille Cedex 20, France

    Patrice Bellot

  2. Virtual University of Tunis, Tunis, Tunisia

    Chiraz Trabelsi

  3. Systèmes d’informations, Big Data et Rec, Institut de Recherche en Informatique de, Toulouse Cedex 04, France

    Josiane Mothe

  4. Department of Computer Science, University of Huddersfield, Huddersfield, United Kingdom

    Fionn Murtagh

  5. DIRO, Universite de Montreal, Montreal, Québec, Canada

    Jian Yun Nie

  6. Pierre and Marie Curie University, Paris Cedex 05, France

    Laure Soulier

  7. Université d'Avignon et des Pays de, Avignon, France

    Eric SanJuan

  8. Department of Information Engineering, University of Padua, Padua, Padova, Italy

    Linda Cappellato

  9. University of Padua, Padua, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ionescu, B. et al. (2018). Overview of ImageCLEF 2018: Challenges, Datasets and Evaluation. In: Bellot, P., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2018. Lecture Notes in Computer Science(), vol 11018. Springer, Cham. https://doi.org/10.1007/978-3-319-98932-7_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-98932-7_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-98931-0

  • Online ISBN: 978-3-319-98932-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation